A repeatable 1 / 2 / 3-job sweep and a report that explains which work remains comfortable.
01 / A fast total can hide a slow job
Suppose three overlapping jobs each produce 200 tokens during the same 10-second window. The combined throughput is 60 tokens per second; each job averaged 20. This is an illustrative calculation, not a TokFire result. The total says how much work the machine processed, while the per-job number better describes one reader’s experience.
Record time to first token, completion latency, per-job generation speed and task success. For a tool agent, also inspect whether the required tools and final facts were correct. A failed task with an impressive generation rate is still a failed task.
02 / Run a controlled concurrency sweep
In TokFire Bench, select one supported model and runtime, then a workload that resembles your intended use. Run one job before selecting two and three concurrent jobs. These are independent tasks calling the same model, not different models loaded together. Keep the model, context, output limits and workload identical.
Use one warm-up and at least three measured rounds per setting. Keep power mode and background activity consistent. Save failed runs as well as good ones. Record the server’s slot and context settings: launching three clients does not prove that the backend actually processes three generations concurrently.
03 / Know which optimization you are testing
Continuous batching lets a serving runtime combine work from active requests. It can improve utilization, but shared compute and memory still limit individual responsiveness. Prompt caching reuses an identical prefix, mainly reducing repeated prompt work. Compare cold and cached prompts separately.
Speculative decoding proposes candidate tokens and has the target model verify them. MTP-capable checkpoints can provide another drafting path in compatible runtimes. Benefit depends on acceptance rate, overhead and memory; it is not a guaranteed multiplier. Keep a baseline without speculation, then compare the same workload with it enabled and document the exact runtime flags.
04 / Grade a use case, not just a machine
Create a results sheet with: jobs; completed/attempted tasks; median first-token time; median and slowest completion; per-job speed; aggregate speed; peak memory; and validation failures. With only a few samples, show the slowest run instead of claiming a statistically stable p95.
Choose acceptance criteria before testing. For example, a foreground assistant may need a short first-response delay; a background report may tolerate a longer wait if all facts are correct. Label those limits as your own targets. A 100–200 tokens-per-second target can be a reference preference, but it is not a verified guarantee for every paid cloud model or plan.