The_DisExcel_project/CLAUDE.md

# The DisExcel Project — Context for Claude Code

Backend: FastAPI + SQLAlchemy (async) + Pydantic v2 + PostgreSQL. JWT auth
(access + refresh tokens), RBAC permissions, Redis cache, RabbitMQ background
workers. Python >=3.13,<4.0, Poetry for dependency management.

## Auth & Permissions

- Users have `direct_permissions` (list of `Permissions`) and `group`
  (list of `PermissionsGroups`, each with its own `permissions`) —
  many-to-many both ways.
- Effective permissions = `direct_permissions ∪ (union of all groups' permissions)`.
- `require_permissions(*permissions)` in `src/web/protected_routes` is a
  FastAPI dependency factory — wraps `CurrentUserService.get_current_user`.
  Call with no args (`require_permissions()`) for "just authenticated, no
  specific permission needed".
- Access tokens carry a `jti` claim. Logout writes `revoked_access_token:{jti}`
  to Redis with TTL = remaining token lifetime — `get_current_user` checks
  this key before anything else.
- `secure` flag on refresh_token cookie is driven by `env_settings.PROD_MODE`
  (bool) — `False` locally/tests so cookies work over plain HTTP, `True` in
  prod.

## Redis (`src/cache/`)

- `RedisClient(redis.Redis)` — module-level shared singleton, subclasses
  `redis.Redis` directly (inherits all commands, no manual wrapping needed).
- Three uses: permissions is-cache was considered and rejected (no real DB
  savings — `get_user_by_id` already eager-loads everything via `selectin`
  in one call); rate limiting on login (`RateLimit.rate_limit(ip)` —
  `INCR` + `EXPIRE` on first attempt, blocks >5/60s); access-token revoke
  blacklist (see above).
- Rate limit is only triggered inside `except HTTPException` on `/protected/token`
  — i.e. only on failed logins, not successful ones (otherwise legitimate
  repeated logins would trip it).

## RabbitMQ (`src/messaging/`)

- `RabbitMQClient` — shared class, lazy `connect()` (can't be async `__init__`),
  holds one `connection` + one `channel`, `get_channel()` ensures setup.
  `connect()` retries `connect_robust()` up to 5 times with exponential
  backoff (`asyncio.sleep(2**attempt)`) before raising — needed because
  when running the app locally (outside Docker Compose) right after
  `docker compose up rabbitmq`, the broker's AMQP listener isn't
  immediately ready and resets the connection mid-handshake
  (`ConnectionResetError`). Compose's own `depends_on: condition:
  service_healthy` only helps when the app itself also runs as a Compose
  service — it does nothing when the app is started on the host.
- **Topology is centralized, not declared ad-hoc.** `src/models/rabbitmq_models/email.py`
  defines the whole topology as pydantic data (`Topology`/`ExchangeSpec`/`QueueSpec`,
  exported as `email_topology`) — topic exchange `"email"`, one durable
  quorum queue per message type (`queue_welcome_email`/`email.welcome`,
  `queue_reset_email`/`email.reset`), each with a matching dead-letter queue.
  `src/messaging/topology_setup.py::apply_topology(channel, topology)` is
  the only place that actually calls `declare_exchange`/`declare_queue`/`bind`
  against RabbitMQ. It's called once at startup — in `main.py`'s `lifespan`
  (web) and in `daemon_run.py`'s `main()` (daemon entrypoint) — **before**
  any producer/consumer touches the exchange/queues. `EmailProducer`/
  `WelcomeEmailConsumer`/`ResetEmailConsumer` no longer declare or bind
  anything themselves — they only `get_exchange("email")` /
  `get_queue("queue_...")`, assuming `apply_topology` already ran.
  Changing a queue's arguments in `email_topology` requires manually
  deleting the existing queue in RabbitMQ first — `declare_queue` on an
  already-existing queue with different arguments fails with
  `PRECONDITION_FAILED`, it doesn't update in place.
- **Dead-lettering**: each original queue's `x-dead-letter-exchange` points
  at `email.dlx`, a **`direct`** exchange (not `fanout` — fanout would
  broadcast every dead-lettered message to *all* DLQs, mixing up
  welcome/reset failures). Each DLQ is bound to `email.dlx` with the same
  routing key as the original queue (`email.welcome`/`email.reset`);
  RabbitMQ preserves a message's original routing key when dead-lettering,
  so no explicit `x-dead-letter-routing-key` override is needed — the
  `direct` exchange alone routes each dead letter back to the correct DLQ.
- `message.process(ignore_processed=True)` + manual `await message.nack(...)`
  in `process_message`, not the plain auto-ack/nack that `message.process()`
  does by default — needed because business-logic exceptions are caught
  *inside* the block (to log them and decide retry vs. dead-letter) rather
  than left to propagate, so `process()` itself must be told not to also
  try to ack/nack on exit. Classification: transient errors (SMTP/network:
  `SMTPServerDisconnected`, `SMTPConnectError`, `TimeoutError`,
  `ConnectionRefusedError`) → `nack(requeue=True)`; everything else
  (bad message body, `SMTPAuthenticationError`, bugs) → `nack(requeue=False)`
  → DLQ. Note `SMTPAuthenticationError` is a known rough edge here — it's
  a systemic config problem (bad credentials), not a per-message one, but
  is currently DLQ'd per-message like any other permanent error rather
  than halting the daemon with an alert.
- Email templates: Jinja2 (`src/service/email/jinja_env.py` — one shared
  `Environment`/`FileSystemLoader` singleton, templates in
  `src/service/email/templates/`), inline CSS (email clients don't support
  `<style>` reliably), `EmailMessage` with `set_content()` (plain-text
  fallback) + `add_alternative(html, subtype="html")`.
- Reset-password flow mirrors welcome: `ResetEmailSender`
  (`src/service/email/email_reset.py`) renders `templates/reset.html`
  (`{{ temp_password }}`, no longer hardcoded) the same way
  `DaemonEmailSender` does `welcome.html`. `ResetEmailConsumer.process_message`
  reads both `email` and `temp_password` from the message body and uses the
  same transient/permanent classification as `WelcomeEmailConsumer`.
  `EmailProducer.send_reset_email(email, temp_password)` takes the password
  as a second argument now. **Still missing**: nothing in the app actually
  calls `send_reset_email` yet — there's no password-reset route that
  generates a `temp_password` and publishes it. Don't assume the
  reset-password feature is reachable end-to-end until that route exists.

## Logging (`src/logging/`)

- All log output (HTTP endpoints, SQL, daemons) funnels through one
  module-level `asyncio.Queue` (`log_queue` in `src/logging/logger.py`) and
  a single `LogWriter.log_writer()` background task that drains it and
  appends to `./logs/{type}_log_{month}_{year}.txt` — this avoids the
  original bug of firing an unsupervised `asyncio.create_task` per log
  line (unordered writes, lost logs if the process died before the task
  ran). **`log_writer()` must be started explicitly wherever the process
  entrypoint is** — it's `asyncio.create_task(writer.log_writer())` in
  `main.py`'s `lifespan` for the web process, and the same call in
  `daemon_run.py`'s `main()` for the daemon process. `log_queue` is
  process-local (plain in-memory `asyncio.Queue`, not shared across
  processes), so **every new entrypoint needs its own writer task** or its
  logs silently queue forever and are never written (unbounded memory
  growth, not a crash).
- `logging.Handler` subclasses (`LoggerDB` for `sqlalchemy.engine`,
  `LoggerDaemon` for the `"daemon"` logger) just push `(type, formatted_msg)`
  onto `log_queue` from `emit()` — do **not** give them a custom `__init__`
  that doesn't call `super().__init__()`; skipping it means `self.level`/
  `self.filters`/etc. never get set and any log call raises `AttributeError:
  'LoggerX' object has no attribute 'level'`.
- Two `ContextVar`s tag log lines with a correlation id without threading
  it through every function call: `request_id_ctx` (set once per HTTP
  request in `LoggingMiddleware.dispatch`) and `message_id_ctx` (meant to
  be set once per RabbitMQ message in `process_message`). Only visible
  within the same async call chain — a `ContextVar` set in an HTTP request
  is `"-"` (the default) inside daemon code, and vice versa; they don't
  cross the process boundary either.
- **`src/logging/logger.py` vs `src/logging/http_logger.py` split matters
  for Docker.** `logger.py` has zero `fastapi`/`starlette` imports —
  intentional, because the `daemon` Poetry group (and therefore the daemon
  Docker image) never installs `fastapi`. `http_logger.py` holds
  `LoggingMiddleware`/`ProcessingTimeMiddleware` (the only things that
  actually need FastAPI/Starlette) and is imported only from `main.py`.
  `src/logging/__init__.py` imports only from `logger.py`. **Never import
  `src.logging.http_logger` from anything that runs in the daemon
  process** (`consumers.py`, `email_welcome.py`, `daemon_run.py`, etc.) —
  it would drag in `fastapi`, which raises `ModuleNotFoundError` in the
  daemon container.
- Plain `logger.exception(...)` must only be called from inside an
  `except` block — it pulls the active exception via `sys.exc_info()` to
  attach a traceback. Called outside `except` (e.g. for a routine "message
  received" log line), it still runs but appends a literal `NoneType: None`
  instead of a traceback, since there's no active exception to format.

## Daemons / workers (`src/daemons/`)

- `BaseDaemon` ABC (`name` + async `run()`), one subclass per consumer
  (`WelcomeEmailDaemon`, `ResetEmailDaemon`, more to come — e.g. reports).
- `DAEMONS` registry dict maps string name → daemon class.
- `daemon_run.py` (project root) is the single entrypoint: `python
  daemon_run.py <name>` runs one daemon, `python daemon_run.py --all` reads
  `configs/daemons.json` (`{"daemons": [...]}`) and runs all enabled ones
  concurrently via `asyncio.gather`. `main()` also applies the RabbitMQ
  topology (`apply_topology`) and starts the log writer task before
  dispatching to either mode, inside a `try`/`finally` so both the
  single-daemon and `--all` paths cancel the writer task on the way out.
- Each daemon runs as its own Docker service/container (`command: ["python",
  "daemon_run.py", "<name>"]`), same pattern as the `migration` service.
- `daemons.json` is read with a plain Pydantic `BaseModel` + manual
  `json.load`, NOT `pydantic-settings` `json_file` — that requires wiring
  `settings_customise_sources` manually in this pydantic-settings version
  and isn't worth the complexity here.

## Docker / Poetry groups

- `pyproject.toml` uses PEP 621 `[project.dependencies]` for shared deps
  (sqlalchemy, redis, aio-pika, pydantic, bcrypt, jose, aiofiles, asyncpg,
  psycopg2-binary, alembic, greenlet).
- `web` group: fastapi, uvicorn, gunicorn, python-multipart — only needed
  by the API server.
- `daemon` group: worker-only deps — `aiosmtpd`, `jinja2` (email sending
  and templating). Must **not** gain a `fastapi`/`starlette` dependency;
  see the `src/logging/logger.py` vs `http_logger.py` split above for why
  that boundary is enforced deliberately.
- `dev` group: pytest stack, allure, httpie, requests-async.
- `claude` group: `graphifyy` (the `/graphify` Claude Code skill) — dev
  tooling only, never referenced by any Dockerfile stage, install with
  `poetry install --with claude` when needed locally.
- Dockerfile has parallel builder→final stage pairs: `builder`→`prod`
  (installs `main,web`) and `worker-builder`→`worker` (installs
  `main,daemon`). Same base pattern: venv builder stage copies
  `/opt/venv` into a clean final stage, poetry itself is uninstalled
  after install to keep the final image lean.
- Alembic runs against a **separate sync engine** (`asyncpg` swapped out,
  psycopg2 used instead) — async SQLAlchemy engine can't drive Alembic
  directly without the `run_sync` bridge, and a dedicated sync engine is
  simpler than that bridge.
- `DB_HOST` differs between contexts: `psql` (Docker service name) for
  containers talking to each other, `localhost` for anything run on the
  host (e.g. local `alembic revision --autogenerate`). Compose services
  override `DB_HOST` via `environment:`; the `.env` file's own default is
  for host-side runs.

## Codebase knowledge graph (graphify)

- `/graphify` (Claude Code skill, `graphifyy` package in the `claude`
  Poetry group) builds a navigable knowledge graph of this repo into
  `graphify-out/` (`graph.html`, `GRAPH_REPORT.md`, `graph.json`) —
  gitignored, regenerable, never commit it.
- Its own interpreter/venv detection defaults to `uv tool`/pipx-style
  installs; **this project has neither** — `graphifyy` is installed
  straight into the project's own `.venv` via Poetry, so point graphify
  at `.venv/bin/python3` directly rather than letting it search for `uv`.
- The sandbox's blanket `**/.env`/`**/.env.*` read-deny blocks graphify
  from reading `configs/.env.example` too (not just the real `.env`) —
  it shows up as `skipped_sensitive` in detection. This is a sandbox-level
  block, not graphify's own sensitive-file heuristic; a real `.env` would
  be excluded by graphify anyway, but `.env.example` (no real secrets)
  would otherwise be safe to include if the sandbox allowed reading it.
- A `GRAPH HEALTH WARNING` (dangling-endpoint edges, collapsed edges) on
  a fresh build most likely means the AST extractor's node-ID format
  changed since a previous partial run — re-run with `graphify extract
  --force` if it persists across rebuilds; a one-off warning on first
  build isn't necessarily a problem.

## Testing (`tests/unit`, `tests/integrated`, `tests/e2e`)

- **Recurring root cause of "different event loop" / `MissingGreenlet`-style
  errors**: prod code uses module-level singletons (`engine`, `redis_client`)
  created once at import time and reused for the app's whole lifetime — this
  is correct for prod (one event loop, whole uptime) but breaks under
  pytest-asyncio's default `function`-scoped event loop (a new loop per
  test, but the singleton's connections stay bound to the *first* loop).
  Fix: test fixtures create a **fresh** `engine`/`RedisClient` per test and
  monkeypatch or inject them in place of the global singleton, then dispose
  on teardown — not a global `session`-scoped event loop (that would mask
  real isolation bugs).
- e2e `MySession` must subclass `httpx.AsyncClient` (not `requests_async.
  AsyncSession` — that library silently drops cookies between requests,
  which broke refresh-token-cookie-dependent tests like logout).
- `test_user_fixture` is `indirect=True` parametrized with
  `(direct_permissions, group)` tuples.
- `tests/unit/test_consumers.py` covers `RabbitMQClient`/`WelcomeEmailConsumer`/
  `ResetEmailConsumer` entirely with mocks — no real broker involved.
  Pattern: `monkeypatch.setattr(rabbitmq_client_module.aio_pika, "connect_robust", ...)`
  patches the module attribute that `connect()` looks up at call time (not
  the `rabbitmq_client` singleton's method — that's a bound method, it has
  no attribute of its own to patch). `aio_pika.connect_robust`/`asyncio.sleep`
  must both be mocked when testing the retry loop, or the test really
  sleeps `2**attempt` seconds between attempts. `message.process(...)` is an
  async context manager, not a plain awaitable — mocking it needs a
  `MagicMock` with `__aenter__`/`__aexit__` set to `AsyncMock`s (see
  `make_fake_message()` in that file), not just `AsyncMock()`. When
  asserting on what a mocked async method returned, compare against
  `mock.return_value` (or a variable captured before assigning it), never
  against the mock itself — `some_mock is some_mock.return_value` is never
  true, they're two different objects.