kmemo-core/dev.kmemo/SemanticCache/getOrPut

getOrPut

suspend fun getOrPut(prompt: String, scope: String = DEFAULT_SCOPE, metadata: Map<String, String> = emptyMap(), compute: suspend (String) -> String): String

Returns the cached answer to prompt, or calls compute and caches what it returns.

The main entry point, and the reason to prefer it over get plus put: the prompt is embedded once and the vector is reused for both the lookup and the write. Doing it by hand costs two embedding calls on every miss.

val answer = cache.getOrPut(prompt) { llm.complete(it) }

Pass context for a turn in a conversation. Without it the cache keys on the last turn alone, so "what about the second one?" can be served an answer computed after a completely different exchange — the context is not weighed, it is ignored. With it, the prior turns are folded into what gets embedded and keyed, so a different conversation is a different question. compute still receives the bare prompt: the context is the cache's business, not the model's.

Concurrent callers asking the same thing are coalesced: the first one computes, the rest wait and are served its answer. Without that, a cold cache under load is worse than no cache — fifty requests for the same prompt arrive together, all miss, and all pay. Coalescing is per scope and per exact prompt text; near-matches still go through the normal lookup. Pass coalesceConcurrentMisses = false to let every caller compute independently.

suspend fun getOrPut(prompt: String, context: List<String>, tags: Set<String> = emptySet(), scope: String = DEFAULT_SCOPE, metadata: Map<String, String> = emptyMap(), compute: suspend (String) -> String): String

The conversation-aware getOrPut: keys the turn on context as well as prompt.

A separate overload rather than a parameter on the one above, because inserting anything ahead of a trailing lambda rebinds it for every caller who passes scope or metadata positionally. That is a source break, and 1.x does not take those.

suspend fun <T> getOrPut(prompt: String, codec: ResponseCodec<T>, scope: String = DEFAULT_SCOPE, metadata: Map<String, String> = emptyMap(), compute: suspend (String) -> T): T

A getOrPut that caches a structured response of type T, not just its text.

The cache still stores a String — codec is what turns your object into one and back — so caching a parsed JSON object, an extracted record, or a tool-call is the same one-embedding round trip as caching text. On a hit the stored text is decoded; on a miss compute runs, its result is encoded and cached, and the same value is returned. See ResponseCodec for the round-trip contract.

val weather: Weather = cache.getOrPut(prompt, weatherCodec) { llm.extractWeather(it) }