Define what the token count contains
Input processing and output generation have different compute behaviour. Long contexts, cache reuse, speculative decoding and batching can change the serving profile. A single output-tokens-per-second number cannot represent every prompt distribution or concurrency level. State whether a price or benchmark covers input, output or both.
Add a service constraint
A fast batch throughput result may be inappropriate for an interactive assistant. Record time to first token, inter-token delay, concurrency and error rate. Quality must also meet the task requirement. A cheap output that fails evaluation is not equivalent to an accepted answer, and retries consume resources even when a customer never sees them.
Use the calculator as a scenario
Our Inference Economics tool multiplies user-entered throughput by provisioned time, serving load and an accepted-output share. It divides the supplied period cost by the resulting accepted output tokens. It does not simulate prefill, batching, queueing or a serving engine. Enter measured throughput for the intended mix; do not substitute a hardware peak.
Close the loop with evaluation
MLCommons provides defined benchmark scenarios; those conditions matter when reading results. CoreWeave’s Forge announcement illustrates the broader industry move toward connected evaluation and iteration. For a procurement decision, define a representative evaluation set, measure the actual workload and preserve both the quality and infrastructure configuration alongside the cost.