SemanticCache

turns prompts into vectors; you supply it.

where entries live and how they expire. Defaults to a bounded in-memory store.

minimum cosine similarity for a candidate to be considered at all. The default is deliberately tight; calibrate it against your own model with dev.kmemo.calibration.ThresholdCalibrator rather than guessing.

vetoes applied to candidates that clear threshold. MatchGuards.standard by default; MatchGuards.none reproduces the naive similarity-only behaviour.

optional final check, typically a cheap model call, run only on candidates that already passed everything else.

optional cap on a single Verifier.verify call. On timeout — or if the verifier throws — the candidate is rejected, not served: a check that could not complete must fail closed, since the verifier exists precisely to keep an unconfirmed answer out. null (the default) applies no timeout.

how many nearest entries to consider. Looking past the closest one matters: when a guard rejects the top candidate, the second may still be a correct answer.

whether concurrent getOrPut calls for the same prompt in the same scope wait for the first one instead of each calling the model. On by default: a cold cache under load is the case where duplicate calls are most expensive and most likely. It governs getOrPutStreaming too, where callers attach to the one provider stream rather than waiting for it, since a streaming caller who waited for the end would be paying the latency they streamed to avoid. getOrPutAll is the exception and says why.

what getOrPut does when the Embedder throws — propagate (the default) or fall back to compute so a lookup is never worse than no cache. See EmbedFailurePolicy. lookup, get and put have no fallback and always propagate. CancellationException always propagates. Every fall-back is counted in CacheStats.degradedLookups and reported as CacheEvent.Degraded, so stepping aside is never silent.

when positive, turns on the exact-match layer: a bounded map from an exact (scope, prompt) to the entry it resolved to, so a byte-for-byte repeat is answered without an Embedder call or a store search. An identical prompt in the same scope is the same question, so the fast path runs no guards and adds no false-hit risk by construction. 0 (the default) keeps it off.

how long a remembered prompt may be served from the exact layer. Set it no longer than your store's TTL: the layer answers without consulting the store, which is what owns expiry and eviction, so a longer window is a window in which it can serve something the store has already dropped. Past the TTL nothing stale is served — the remembered embedding is still reused, so the lookup goes through the ordinary path with the network call already paid. null means the layer never expires, which is only correct when the store has no TTL either.

per-scope overrides of threshold, consulted by scope name with the global value as the fallback. One threshold is necessarily wrong for a service answering both regulated and casual questions, and tuning the single value to the strictest caller makes every other caller pay for it.

when non-empty, puts the cache in shadow mode: every getOrPut runs the full lookup, reports what it would have decided at each of these thresholds through CacheEvent.Shadow, and then always computes. Nothing is ever served from the cache, so a false hit cannot reach a user while you are still choosing a threshold; writes still happen, because a shadow cache that never fills would report a miss for everything and measure nothing.

reorders the candidates that cleared the threshold before the guards see them, or null (the default) to try them nearest-first. It reorders and never rescores, and it runs after the threshold filter, so it can change the order the cache tries candidates in but never which candidates are eligible. MmrReranker is the one that ships; what it buys is fewer Verifier calls on a cache whose nearest entries are near-duplicates of each other.

when non-null, a similarity at or above which a new entry replaces the existing entry it duplicates instead of joining it. A cache that has answered the same question in six phrasings stores six copies of one answer, and every later lookup pays to score all six. Only an entry that clears this similarity and passes every guard is replaced, so deduplication can never merge two entries the cache would have refused to serve for each other. null (the default) keeps every write. Costs one extra store search per write, on the write path rather than the read path.

when non-null, lets each scope's threshold follow its own traffic instead of staying where it was configured. Requires a verifier, and the constructor throws without one: adaptation lowers the threshold as well as raising it, and the only thing that makes lowering safe is something above the threshold that can tell a right answer from a wrong one. Pass the same object in listeners so it can see the outcomes it adapts on. See AdaptiveThresholds.

refuses any read or write that did not come through forTenant. Off by default, because a single-tenant cache has nothing to isolate. On, it turns "somebody forgot the tenant" from a hit belonging to another customer into a failure at the call site, which is the only form in which an isolation property is worth having. See TenantedCache.

when non-null, makes a prompt earn its place before its answer is stored: entries are written on the second sighting of the exact prompt rather than the first, so a store in front of real traffic stops filling with questions asked once at three in the morning. Off by default. It can only ever suppress a write, never a lookup, so a wrong decision costs a future miss and nothing else. Applies to the write that follows a miss, not to put or warm. See AdmissionPolicy for what it costs in hit rate.

what a model call costs, per scope, so the cache can report what its hits saved rather than leaving a hit count to be multiplied by an average nobody has. Empty by default and empty is honest: this library ships no table of provider prices, because one would be wrong the month after it shipped. The token counts are read from the served entry's CacheEntry.metadata by the keys TokenPrices names, so a saving is the cost of the call that was actually avoided. Reported in CacheStats.savings and on CacheEvent.Hit.saved. See TokenPrices.

vetoes writes of data that must never be persisted, consulted once per write on every write path including warm. null (the default) caches everything the cache decides to cache. A vetoed write is a policy decision, not a failure: the call still returns its response. See CachePolicy.

when positive, turns on a bounded negative cache: the embedding of a prompt that just missed is remembered, so an immediate repeat of the same brand-new prompt is embedded once rather than once per caller. Extends the concurrent-miss coalescing to the near-in-time sequential case. It only ever reuses an embedding — it never suppresses the store search — so it cannot cause a false hit. 0 (the default) keeps it off.

how long a remembered miss stays usable when negativeCacheSize is positive, or null to keep it until evicted. A short TTL is the point: it should cover a burst, not pin a stale embedding for a prompt that has since been answered elsewhere.

observers notified of every hit, miss and write as it happens (see CacheEvent). Empty by default, and an empty list is free: with no listeners the cache builds no events and measures no latencies, so the hot path is exactly as it was. Each listener runs inline and must be fast and non-throwing — see CacheListener.

when non-null, turns on write-behind: on a getOrPut miss the cache returns as soon as compute does and the store write is applied off the caller's critical path, by a single worker running on this scope. Writes are applied in submission order while buffered; if the buffer is full the write falls through synchronously rather than being dropped, so a write is never lost (only, rarely, reordered under saturation). The window between compute and write is a small chance of a duplicate compute for the same brand-new prompt. Cancel this scope to stop the worker. null (the default) writes through synchronously, and put/warm always do.

how many pending writes to buffer before falling through to a synchronous write. Only meaningful when writeBehindScope is set.

time source for entry timestamps.