AdaptiveThresholds

Moves each scope's similarity threshold towards the value its own traffic justifies.

When this is safe, which is the only part that matters

A threshold decides what gets served. Moving it automatically on unlabelled traffic is guessing with correctness, and this library does not do that — a cache has no way to know that a hit it served was right. So adaptation is refused outright unless a Verifier is configured, and SemanticCache throws at construction rather than quietly running without one.

With a verifier in the loop the threshold stops being a correctness knob and becomes a cost knob. Everything above it is checked by something that can tell right from wrong; the threshold only decides how many candidates get that far, and therefore how many model calls the verifier makes. That is a quantity a cache can observe about itself, and it is what this adapts on.

What it reads

Per scope, the outcome of every candidate that reached the verifier: served, or refused. Two rules, and both point the same way:

• The verifier refusing more than targetRejectionRate of what it sees means the threshold is letting through candidates that were never going to be served. Raise it, and pay for fewer calls.

• The verifier refusing far less than that means the threshold is stricter than the verifier needs it to be, and entries that would have been served are not being looked at. Lower it.

The second rule is the one that would be reckless without a verifier, and is why the requirement is not negotiable.

What it deliberately does not do

It does not move on a handful of lookups: nothing happens in a scope until minimumSamples candidates have been verified there, and the window resets after each move so the next decision is made on the traffic that followed it. It never leaves [floor, ceiling], which are yours to set. And it adapts one step at a time rather than jumping to a computed optimum, because the measurement is of the traffic that arrived under the current threshold and says nothing about traffic three steps away.

val adaptive = AdaptiveThresholds(floor = 0.86, ceiling = 0.97)
val cache = SemanticCache(
    embedder = embedder,
    verifier = verifier,
    adaptiveThresholds = adaptive,
    listeners = listOf(adaptive),
)

The same object is the listener and the source: it has to see the outcomes to have an opinion about them. Passing it only as a listener gives you recommendationFor to read without the cache acting on it, which is the way to watch what it would do before letting it.

Parameters