Every AI tool pitch now includes a performance chart. A tall bar on the left, a short bar on the right, a multiplier in bold. Teams buy on that chart, then discover in production that their workload looks nothing like the one that was measured.
The useful question for an operator is not "how fast can it go?" It is "where does the speed come from, and what does each piece cost to keep?"
Where the Speed Actually Comes From
The PyTorch team published one of the most transparent breakdowns available in Accelerating Generative AI Part III: Diffusion, Fast. They took Stable Diffusion XL at 30 steps and added one optimisation at a time:
| Optimisation | Latency per image | Operational cost |
|---|---|---|
| Baseline | 7.36 s | n/a |
| Reduced precision (bfloat16) | 4.63 s | Near zero |
| Efficient attention (SDPA) | 3.31 s | Near zero; on by default in PyTorch 2 |
| torch.compile | 2.54 s | Compile time; recompiles when inputs change shape |
| Fused projections | 2.52 s | Experimental support |
| Dynamic int8 quantization | 2.43 s | Extra library; can slow small workloads |
The pattern matters more than the multiplier. Two low-cost settings removed 55% of the latency. Everything after that fought over the last 0.88 seconds, and each step added something to maintain. The run used a 400 W, 80 GB A100 with its clock set to maximum, which is not the hardware most teams deploy on.
The Costs a Benchmark Cannot Show
The Diffusers acceleration guide states the catch plainly: compilation is slow the first time, and running the compiled pipeline on a different image size "retriggers compilation which is slow and inefficient." A benchmark renders one size hundreds of times. A creative team changes formats constantly. The headline gain and the realised gain are different numbers.
The Diffusers memory guide adds that some layout optimisations "may result in worst performance" on operators that do not support them. Quantization bought 4% in the published run and needs a dependency your environment may not carry.
Why Vendors Lead With the Multiplier
A multiplier is the easiest number in technology marketing to make true and the hardest to make useful. It needs only two measurements, a slow baseline and a fast result, and the vendor chooses both. Start from full precision with efficient attention switched off, as the PyTorch baseline deliberately did to show each step, and almost any modern stack looks like a dramatic improvement.
That is not dishonest in a research post that shows every step. It becomes misleading when the baseline disappears and only the multiplier survives into a sales deck. "Three times faster" than a configuration nobody runs in production is a statement about the baseline, not about the product.
The defence is simple: always ask "faster than what?" If the answer is a setup your team would never use, discount the claim to the gain over your actual current configuration. In many cases that gain is a fraction of the headline, because your current stack already has the cheap wins.
The Line Item Nobody Prices: Failure
Performance work has a quiet failure mode. The more aggressive the configuration, the narrower the range of hardware and inputs it tolerates. A setting that is fastest on a 24 GB workstation can run out of memory on a 12 GB laptop. A compiled pipeline that flies at one image size stalls while it recompiles at another.
For a business, a failed render is not neutral. It costs the waiting time, the retry, and the attention of the person who has to notice and restart it. Multiply that across a team and a fleet of mixed machines, and an "optimisation" that fails 5% of the time can cost more than it saves. Reliability belongs in the performance budget, not next to it.
Four Questions for Any Performance Claim
- On what hardware? A data-centre GPU result is a hypothesis about your laptop fleet, not a result.
- On what workload shape? One fixed size, or the mix your team actually produces?
- Which gains are defaults? If most of the speed is precision and attention, a competitor probably has it too.
- What happens when it fails? A fast configuration that crashes on low-memory machines has negative value.
What Good Looks Like
We built this thinking into Vision Studio, our open-source local AI generation app. Its Performance panel sets six optimisations to Auto, On or Off behind one master switch, and reports which were applied, skipped or fell back, so a setting that silently did nothing never looks like one that worked. Two options that need packages the standard build does not ship stay off there rather than failing quietly. When memory runs short, the engine steps down a fallback ladder instead of failing the job.
That is the pattern to demand from any vendor: performance that is explained, reported and recoverable, not a multiplier on a slide.
Run Your Own Evaluation in a Week
- Pick three real jobs from last month's work, at the sizes and formats your team actually used.
- Run them on your lowest and highest specification machines, not only the best one.
- Record time to an approved result, including retries and failures, not time per successful render.
- Change one setting at a time, so you know which change bought which gain.
A week of this produces a number you can defend in a budget meeting, which is more than any vendor chart can offer.
The Takeaway for Leaders
- Most AI speed comes from settings that are cheap and widely available. Do not pay a premium for them.
- The expensive optimisations have operational costs. Price those in before you price in the gain.
- Require vendors to show results on your hardware and your workload mix.
- Reliability is part of performance. A job that finishes slower beats a job that does not finish.
The full technical breakdown, including the Performance panel design is on the Vision Studio blog at vision-studio-x.com/blog/fastest-setting-does-not-crash. Vision Studio is a free, open-source, local-first AI generation app built by Strategia-X. If you want a second pair of eyes on how your organisation adopts AI tooling, talk to us at strategia-x.com.
-Rocky
#AIPerformance #TechnologyStrategy #TCO #PyTorch #LocalAI #OpenSource #StrategiaX #VisionStudio