ThresholdCalibrator

Finds the similarity threshold that fits your embedding model, by measuring instead of guessing.

A threshold copied from a blog post is a threshold tuned to somebody else's model. The same pair of prompts might score 0.86 with one embedding model and 0.94 with another, so a value that is safely tight for one is dangerously loose for the next — and the failure is silent, because a false hit looks exactly like a fast response.

Give the calibrator prompt pairs you have labelled and it sweeps the threshold range, reporting how many correct hits and how many wrong answers each setting produces on your own model, with your own guards in the loop:

val report = ThresholdCalibrator(myEmbedder).calibrate(myLabelledPairs)
println(report.summary())
val cache = SemanticCache(myEmbedder, threshold = report.recommended.threshold)

A hundred pairs drawn from your own traffic is plenty, and half of them should be near-misses: prompts that look alike but need different answers. Pairs that are obviously unrelated teach the calibrator nothing, because no threshold ever confuses them.

Parameters