SemanticCache
A cache keyed by what a prompt means rather than by its exact bytes.
Every prompt is embedded, compared against the prompts already cached, and — if something is close enough and survives the guards — answered from the cache instead of from the model. Two prompts worded differently but asking the same thing hit the same entry, which an exact-match cache can never do.
val cache = SemanticCache(
embedder = myEmbedder,
store = InMemoryStore(maxEntries = 10_000, ttl = 1.hours),
)
val answer = cache.getOrPut(prompt) { llm.complete(it) }Why this is not just a threshold
The failure mode of a semantic cache is not a miss, it is a false hit: returning a cached answer to a question it does not answer. Convert 100 USD to EUR and Convert 250 USD to EUR embed at around 0.99 with every mainstream model. There is no threshold that accepts real paraphrases and rejects that pair, because on the similarity axis the pair is closer than most paraphrases are.
So similarity is only the first filter here. Candidates that clear threshold are then read as text by a chain of MatchGuards looking for concrete evidence that the answers must differ — a different number, unit, entity, time reference, or a flipped comparison. Anything that survives can be sent to an optional Verifier for a final check. The costs are asymmetric and the defaults follow that: a wrong rejection costs one API call, a wrong acceptance costs a wrong answer.
Scopes
Every entry belongs to a scope, and lookups only see their own. Anything that changes what a correct answer looks like — model, temperature, system prompt, tenant, user's language — belongs in the scope string, otherwise the cache will happily serve one model's answer to another's caller:
cache.getOrPut(prompt, scope = "gpt-4o|t=0.0|v3") { llm.complete(it) }Instances are safe to share across coroutines, as long as the Embedder and CacheStore are.
Parameters
turns prompts into vectors; you supply it.
where entries live and how they expire. Defaults to a bounded in-memory store.
minimum cosine similarity for a candidate to be considered at all. The default is deliberately tight; calibrate it against your own model with dev.kmemo.calibration.ThresholdCalibrator rather than guessing.
vetoes applied to candidates that clear threshold. MatchGuards.standard by default; MatchGuards.none reproduces the naive similarity-only behaviour.
optional final check, typically a cheap model call, run only on candidates that already passed everything else.
optional cap on a single Verifier.verify call. On timeout — or if the verifier throws — the candidate is rejected, not served: a check that could not complete must fail closed, since the verifier exists precisely to keep an unconfirmed answer out. null (the default) applies no timeout.
how many nearest entries to consider. Looking past the closest one matters: when a guard rejects the top candidate, the second may still be a correct answer.
whether concurrent getOrPut calls for the same prompt in the same scope wait for the first one instead of each calling the model. On by default: a cold cache under load is the case where duplicate calls are most expensive and most likely.
what getOrPut does when the Embedder throws — propagate (the default) or fall back to compute so a lookup is never worse than no cache. See EmbedFailurePolicy. lookup, get and put have no fallback and always propagate. CancellationException always propagates.
when positive, turns on a bounded negative cache: the embedding of a prompt that just missed is remembered, so an immediate repeat of the same brand-new prompt is embedded once rather than once per caller. Extends the concurrent-miss coalescing to the near-in-time sequential case. It only ever reuses an embedding — it never suppresses the store search — so it cannot cause a false hit. 0 (the default) keeps it off.
how long a remembered miss stays usable when negativeCacheSize is positive, or null to keep it until evicted. A short TTL is the point: it should cover a burst, not pin a stale embedding for a prompt that has since been answered elsewhere.
observers notified of every hit, miss and write as it happens (see CacheEvent). Empty by default, and an empty list is free: with no listeners the cache builds no events and measures no latencies, so the hot path is exactly as it was. Each listener runs inline and must be fast and non-throwing — see CacheListener.
when non-null, turns on write-behind: on a getOrPut miss the cache returns as soon as compute does and the store write is applied off the caller's critical path, by a single worker running on this scope. Writes are applied in submission order while buffered; if the buffer is full the write falls through synchronously rather than being dropped, so a write is never lost (only, rarely, reordered under saturation). The window between compute and write is a small chance of a duplicate compute for the same brand-new prompt. Cancel this scope to stop the worker. null (the default) writes through synchronously, and put/warm always do.
how many pending writes to buffer before falling through to a synchronous write. Only meaningful when writeBehindScope is set.
time source for entry timestamps.
Constructors
Functions
A getOrPut for streaming completions: caches the assembled text of a streamed answer and replays it on a hit, so a streaming caller never has to fall back to the blocking path.
Removes the entry id, typically one reported by a CacheLookup.Hit that proved wrong.
Counters since this instance was created. See CacheStats for what they tell you.