inference-obs

The problem

When several of your own projects start calling models, three problems appear at once: you do not know how much each one spends, you do not know why a response came out wrong, and every project ends up with its own key and its own mess.

inference-obs solves that with a single entry point: an OpenAI-compatible proxy that enforces a per-project budget, and a trace server that stores every call with its attribution.

Two applications, one operation

The core is LiteLLM with PostgreSQL in a container under supervisord, and it serves chat and transcription. The observability is Arize Phoenix, which receives traces over the open protocol and also hosts the datasets and the evaluation experiments.

It lives in an environment with 4 GiB of memory, so the envelope is not an optimisation detail: it is a requirement. The deployment plan allocates the core and the observability within that budget, with an explicit memory limit for the main container, and the check is written down.

What matters most to me about this project

That observability can go down without taking inference with it. That is the central architecture decision: the core sends traces over public HTTPS, so if Phoenix does not respond, the trace is lost and the call goes on. It is verified with a test that brings up both services, kills the tracing one and checks that inference still returns 200.

The opposite —losing traces taking the service down— is a priority mistake that gets made by default when observability is treated as part of the critical path.

And that the evaluation tells the truth about itself. The model comparator reports separately the failures of the task, those of the automatic judge, and the judgements that could not be issued. A single average would have hidden exactly what needed to be known.