LLM API Integration Rules

An LLM API call is an external, non-deterministic dependency — build the integration the way you'd build around any unreliable third-party service: validate its output, expect failures, and never let it silently corrupt downstream state.

Never trust model output blindly

• Validate and parse model output defensively before using it — for structured output (JSON, function-call arguments), parse with a schema validator and handle the malformed case explicitly rather than assuming the model always returns valid structure.
• Never pass model-generated content directly into a sensitive sink (a SQL query, a shell command, a URL that will be fetched, rendered HTML) without the same sanitization/parameterization you'd apply to any untrusted user input — the model can be manipulated via its input (prompt injection) into producing adversarial output.
• Treat factual claims in model output as unverified unless the output is explicitly grounded in retrieved/provided source data — a fluent, confident-sounding answer is not evidence of correctness.

Streaming UX

• Stream tokens to the user as they're generated for any response likely to take more than a second or two — perceived latency drops sharply with streaming even when total generation time is unchanged.
• Handle a stream that ends abnormally (connection drop, provider error mid-stream) by showing the partial output plus a clear error/retry affordance, not by silently discarding what was already generated.
• Debounce or batch UI updates from a token stream (e.g. render every N tokens or on animation frame) rather than triggering a full re-render per token — naive per-token rendering can visibly jank on longer outputs.

Retry with backoff on rate limits

• Retry on rate-limit (429) and transient server errors (5xx) with exponential backoff and jitter, not a fixed retry interval — a fixed interval across many concurrent clients causes retry storms that make the rate limit worse.
• Respect a Retry-After header when the provider sends one, rather than using a client-side backoff guess — the provider knows its own recovery window.
• Cap retry attempts and surface a clear failure to the caller after the cap is hit — infinite or unbounded retries turn a transient provider issue into a hung request from the user's perspective.

Token budget management

• Track and cap the token count of assembled prompts (system + context + history + user input) before sending — an unbounded prompt silently grows until it hits the model's context limit and fails, often at the worst possible moment (a long conversation).
• Truncate or summarize conversation history deliberately (drop oldest turns, summarize older context) when approaching the budget, rather than letting the API call fail on overflow and only then reacting.
• Budget separately for input and expected output tokens — a prompt that fits the context window can still fail if there isn't enough headroom left for the response.

Prompt and response logging with PII care

• Log prompts and responses for debugging and quality monitoring, but treat that log store with the same access controls and retention policy as any other store containing user content — LLM logs routinely contain names, contact details, and other PII typed by users.
• Redact or avoid logging known-sensitive fields (payment details, credentials, health information) before they ever reach the prompt, not just before logging — the safest redaction happens at the point of prompt assembly.
• Set an explicit retention/deletion policy for prompt/response logs rather than keeping them indefinitely by default.

Fallback behavior on failure

• Define an explicit fallback for every LLM-dependent user-facing feature: a cached previous result, a simpler non-LLM code path, or a clear "temporarily unavailable" state — a feature that just breaks when the provider is down is a worse experience than a degraded but functional one.
• Never let an LLM call failure cascade into failing an entire request/page when the LLM output is enhancement rather than core functionality — isolate it so the rest of the page/response still succeeds.

Cost monitoring

• Track token usage and cost per feature/endpoint, not just in aggregate — aggregate cost alone doesn't reveal which feature or which misbehaving prompt is driving spend.
• Set alerting thresholds on unexpected cost/usage spikes — a bug that causes retry loops or unbounded context growth can silently multiply cost before anyone notices from the bill alone.

Model version pinning

• Pin to a specific model version in production rather than an auto-updating "latest" alias — providers periodically change default model behavior, and an unpinned alias can silently shift output quality, format, or cost without a corresponding code change to explain it.
• Treat a model version upgrade as a deliberate, tested change (re-run eval/test cases against the new version) rather than an automatic background update.