---
title: Production error analysis — 4 September 2026
description: A postmortem of five failed Qgen requests: one 429 on the RAG selection call took down every model in the fallback sequence.
date: 2026-09-05
lang: en-US
author: Julien Béranger
model: Claude Opus 5
source: https://julienberanger.com/qgen-error-analysis-sept-4-2026
---

# Production error analysis — 4 September 2026

Source: `/home/ubuntu/.pm2/logs/qgen-error.log` (last 500 lines).

## Summary

Five requests against a qgen context failed end to end (15:05, 15:07,
15:08, 15:14 and 21:46 UTC), each ending in an unhandled
`TypeError: Cannot read properties of undefined (reading 'substring')`.
A single transient Mistral rate limit was enough to make every model in the
fallback sequence fail, and the crash then hid the failure from the caller.

## The cascade

1. **Rate limit on the RAG selection call.**
   `RagService.selectRelevantFiles` calls `mistral-small-latest` to pick which
   context files are relevant. Mistral answered
   `429 {"message":"Rate limit exceeded","code":"1300"}`.

   ```
   ERROR [MistralService] Mistral API error response: {"object":"error","message":"Rate limit exceeded",...}
   ERROR [RagService] Error in file selection for context qgen: Failed to process message with Mistral AI (mistral-small-latest)
   ```

2. **The fallback loaded the entire context.** On selection failure the service
   returned *every* file in the index. The context directory is ~4.8 MB, so
   the system prompt became roughly 2 million tokens — larger than any
   configured model's context window. What was meant as graceful degradation was
   in fact a guaranteed failure.

3. **Every model in the fallback sequence rejected the prompt**, each for its own
   reason, so retrying across providers could not help:

   | Model | Error |
   |---|---|
   | `anthropic` | `prompt is too long: 2250703 tokens > 200000 maximum` |
   | `mistral` | `403 tier_not_allowed` — "This model is not available in your subscription tier" |
   | `openai` | `429 rate_limit_exceeded` — TPM limit 200 000, requested 1 205 848 |
   | `deepseek` | `maximum context length is 1048576 tokens, requested 2008311` |

   ```
   ERROR [AppService] All models in fallback sequence failed. Last error: Failed to process message with DeepSeek
   ```

4. **The failure crashed the logger instead of being reported.** With no model
   succeeding, `output` stayed `undefined`; `JSON.stringify(undefined)` returns
   `undefined`, and the log-append step called `.substring(0, 10)` on it:

   ```
   ERROR [AppService] TypeError: Cannot read properties of undefined (reading 'substring')
       at AppService.ask (/home/ubuntu/qgen/dist/app.service.js:542:35)
   ```

   The outer `try/catch` in `ask()` then swallowed the exception and returned
   **HTTP 200 with `output: undefined`** — the client saw a successful-looking
   empty response rather than an error.

## Root causes

- **Unbounded RAG fallback.** Falling back to "all files" is only viable for
  small contexts. At 4.8 MB it turned a recoverable 429 into a total outage for
  that request.
- **No retry on the selection call.** A single 429 on a cheap, fast call took
  down the whole request path; one retry with backoff would have avoided all
  five incidents.
- **Failure treated as success.** `ask()` had no error path for "no model
  produced output", so the fault surfaced as a crash in unrelated logging code
  and then as a silent 200.

## Fixes applied

- `src/rag/rag.service.ts` — the selection-failure fallback now returns
  `RAG_REQUIRED_FILES` plus at most `RAG_MAX_FILES` other files (and logs the
  degradation), instead of the whole context.
- `src/app.service.ts` — when no model succeeds, `ask()` throws
  `ServiceUnavailableException` carrying the last provider error;
  `HttpException`s are re-thrown from the outer catch instead of being converted
  into a 200. The log-append call is also guarded against a non-string output.
- `src/app.service.spec.ts` — the test asserting the old "return an empty
  response" contract now asserts the exception.

## Still open

- **No retry/backoff** on the `mistral-small-latest` selection call.
- **`mistral` is unusable in the fallback sequence** — the configured large model
  returns `403 tier_not_allowed` on the current subscription, so that slot
  always fails.
- **`docs/FALLBACKS.md` is now stale**: it still documents "system loads ALL
  files from the context" as the RAG selection fallback.

## Other errors visible in the same log

- **`OFF_TOPIC` logged as ERROR** (~50 occurrences, April–September). This is
  normal control flow — the off-topic guardrail firing — logged at error level.
  Pure noise; belongs at `warn` or `debug`.
- **`Cannot find module '/home/ubuntu/qgen/dist/main'`** (repeated, earliest
  entries). PM2 restart-looping against a missing build — the service was
  started before `pnpm build` had produced `dist/`.
- **Anthropic `invalid_request_error`: "Your credit balance is too low"**
  (7 July, ~12 requests). Billing exhaustion, since resolved; note that requests
  did *not* fall through to another provider cleanly at the time.