· 3 min read · AI Engineering
Observability for LLM Features Without Drowning in Logs
You need to know what your AI feature did, what it cost, and whether it worked, without storing every user's words forever.
Log metadata by default: model and prompt versions, latency, token counts, cost, and outcome signals. Store full prompts and outputs only sampled, briefly, and never with secrets or personal data.
Four numbers catch most failures
Start with latency, error rate, cost per request, and an outcome signal: did the user accept, edit, or abandon the result? Those four series, split by model and prompt version, catch the failures that actually hurt: the provider slowdown at peak hour, the prompt change that doubled token use, the release users quietly stopped trusting.
Version tags are what make the graphs actionable. An untagged latency spike is weather. A spike that begins exactly at prompt version 14 is a rollback decision you can make in minutes.
Full-text logs are a liability with a search box
It is tempting to log every prompt and completion for debugging. It is also how user pasted secrets, personal details, and occasionally credentials end up retained in a system nobody audited. Treat model traffic like payment traffic: keep the receipt, not the contents.
When you truly need examples, sample deliberately, scrub before storage, expire quickly, and say so in your retention policy. A small, consented debug set beats an accidental archive of everything everyone ever typed.
“Observe the behavior of the system, not the private words of the people using it.”
Alerts people can act on
Alert on user-visible symptoms: error budget burning, acceptance rate dropping, cost per success doubling. Page a human for those. Leave the token-count trivia in dashboards for the morning review, or your team will learn to ignore the pager, which is the most expensive lesson in operations.
Good observability for AI is mostly good observability, full stop. The models change. The discipline does not.