Production error analysis — 4 September 2026

Source: /home/ubuntu/.pm2/logs/qgen-error.log (last 500 lines).

Summary

Five requests against a qgen context failed end to end (15:05, 15:07, 15:08, 15:14 and 21:46 UTC), each ending in an unhandled TypeError: Cannot read properties of undefined (reading 'substring'). A single transient Mistral rate limit was enough to make every model in the fallback sequence fail, and the crash then hid the failure from the caller.

The cascade

1. Rate limit on the RAG selection call. RagService.selectRelevantFiles calls mistral-small-latest to pick which context files are relevant. Mistral answered 429 {"message":"Rate limit exceeded","code":"1300"}.

   ERROR [MistralService] Mistral API error response: {"object":"error","message":"Rate limit exceeded",...}
   ERROR [RagService] Error in file selection for context qgen: Failed to process message with Mistral AI (mistral-small-latest)

2. The fallback loaded the entire context. On selection failure the service returned every file in the index. The context directory is ~4.8 MB, so the system prompt became roughly 2 million tokens — larger than any configured model's context window. What was meant as graceful degradation was in fact a guaranteed failure.

3. Every model in the fallback sequence rejected the prompt, each for its own reason, so retrying across providers could not help:

   Model	Error
   anthropic	prompt is too long: 2250703 tokens > 200000 maximum
   mistral	403 tier_not_allowed — "This model is not available in your subscription tier"
   openai	429 rate_limit_exceeded — TPM limit 200 000, requested 1 205 848
   deepseek	maximum context length is 1048576 tokens, requested 2008311

   ERROR [AppService] All models in fallback sequence failed. Last error: Failed to process message with DeepSeek

4. The failure crashed the logger instead of being reported. With no model succeeding, output stayed undefined; JSON.stringify(undefined) returns undefined, and the log-append step called .substring(0, 10) on it:

   ERROR [AppService] TypeError: Cannot read properties of undefined (reading 'substring')
       at AppService.ask (/home/ubuntu/qgen/dist/app.service.js:542:35)

   The outer try/catch in ask() then swallowed the exception and returned HTTP 200 with output: undefined — the client saw a successful-looking empty response rather than an error.

Root causes

• Unbounded RAG fallback. Falling back to "all files" is only viable for small contexts. At 4.8 MB it turned a recoverable 429 into a total outage for that request.
• No retry on the selection call. A single 429 on a cheap, fast call took down the whole request path; one retry with backoff would have avoided all five incidents.
• Failure treated as success. ask() had no error path for "no model produced output", so the fault surfaced as a crash in unrelated logging code and then as a silent 200.

Fixes applied

• src/rag/rag.service.ts — the selection-failure fallback now returns RAG_REQUIRED_FILES plus at most RAG_MAX_FILES other files (and logs the degradation), instead of the whole context.
• src/app.service.ts — when no model succeeds, ask() throws ServiceUnavailableException carrying the last provider error; HttpExceptions are re-thrown from the outer catch instead of being converted into a 200. The log-append call is also guarded against a non-string output.
• src/app.service.spec.ts — the test asserting the old "return an empty response" contract now asserts the exception.

Still open

• No retry/backoff on the mistral-small-latest selection call.
• mistral is unusable in the fallback sequence — the configured large model returns 403 tier_not_allowed on the current subscription, so that slot always fails.
• docs/FALLBACKS.md is now stale: it still documents "system loads ALL files from the context" as the RAG selection fallback.

Other errors visible in the same log

• OFF_TOPIC logged as ERROR (~50 occurrences, April–September). This is normal control flow — the off-topic guardrail firing — logged at error level. Pure noise; belongs at warn or debug.
• Cannot find module '/home/ubuntu/qgen/dist/main' (repeated, earliest entries). PM2 restart-looping against a missing build — the service was started before pnpm build had produced dist/.
• Anthropic invalid_request_error: "Your credit balance is too low" (7 July, ~12 requests). Billing exhaustion, since resolved; note that requests did not fall through to another provider cleanly at the time.