A bound is not a constant somebody picked. It is derived from what the manifest says was done to the model, and this page is the argument for each number.
|produced - reference| <= atol + rtol * |reference|, per element, which is numpy’s allclose and
is used here for the same reason: neither half works alone.
A pure relative bound is meaningless wherever the reference crosses zero, and a normalize step that subtracts a mean guarantees it will. A pure absolute bound is sized by the largest value in the tensor, so it excuses everything small.
Two axes, and they compound.
The recipe says how the weights were stored. A full-precision model should agree with its
reference to within float32 rounding. An int8-dynamic model should not, and holding it to the first
bound would fail a model doing exactly what it was told.
The precision says what the delegate does arithmetic in, which is a different question. Core ML and MPS run at float16 by default, so an artifact lowered for either answers to a different bound even when nothing was quantized. Without this recorded, the gate failed models that were correct.
They compound because an int8 model on a half-precision GPU is wrong in both ways at once, so
boundFor takes the larger of the two contributions per axis rather than the first one it
finds.
Float16’s machine epsilon is 4.9e-4, and a bound of a small multiple of that is the number that
looks right. It is wrong, and the reason is worth stating because it generalises.
A gate sees outputs. Rounding happens on intermediates. Feed this project’s own model an input of
magnitude 1e3 and its intermediate activations sit around 1e3 while its outputs come out near
9.4, because the last layer subtracts two numbers of similar size. The absolute error accumulated
at 1e3 survives the cancellation; the magnitude it is measured against does not. Divide one by the
other and the relative error at the output is two orders larger than epsilon at the intermediate.
So float16 sits at 2e-2 relative. That is not slack, it is the arithmetic.
Every bound in the table except one comes from a measurement, and each entry cites what it was
measured against. packages/fluttorch_executorch/tool/measure_tolerances.dart reproduces them.
Two models, and the difference between them is the part worth reading. testdata/matrix is a
convolutional network with two kinds of normalisation and a softmax; testdata/taps and its
siblings are the two-layer model. The simpler one drifts up to eight times further on every backend,
because its outputs land near 9.4 while the other ends in a softmax that pins its own into
[0, 1], and a relative error is measured against the output while the rounding happened on
intermediates. The bounds are set by the model with the large outputs. Narrowing float16 to what the
convolutional model alone would justify was tried and failed the other fixture immediately.
The exception is int4-weight-only, which is still a starting point and says so in its own comment:
nothing in this toolchain exports int4, so there is no artifact to measure.
None of that makes a bound an answer about your model. The number that is right for one is a property of its activation ranges, which is exactly what the two models above demonstrate. Measure yours.
You can pass your own, and the gate will use it:
await expectParity(model, goldens: goldens, tolerance: myTolerance);
Doing that is sometimes right and is worth being suspicious of. A bound loosened until the build goes green is a bound that measures nothing, and the failure it was hiding is still there. If a model needs a wider bound than its recipe implies, the interesting question is why, and the report names the tensor and the backend so that question has somewhere to start.
A report says which tensor moved, by how much, against which bound, and on which backend. It says that whether the answer is pass or fail, which is deliberate: a run that passed by a hair and one that passed comfortably are different situations, and a boolean cannot tell them apart.
Where the export captured activation taps, the report also names the earliest layer whose numbers moved. A drift at the output tells you the model is wrong. A drift attributed to the third layer tells you where to look.