getOrPut

Returns the cached answer to prompt, or calls compute and caches what it returns.

The main entry point, and the reason to prefer it over get plus put: the prompt is embedded once and the vector is reused for both the lookup and the write. Doing it by hand costs two embedding calls on every miss.

val answer = cache.getOrPut(prompt) { llm.complete(it) }

Concurrent callers asking the same thing are coalesced: the first one computes, the rest wait and are served its answer. Without that, a cold cache under load is worse than no cache — fifty requests for the same prompt arrive together, all miss, and all pay. Coalescing is per scope and per exact prompt text; near-matches still go through the normal lookup. Pass coalesceConcurrentMisses = false to let every caller compute independently.

A getOrPut that caches a structured response of type T, not just its text.

The cache still stores a String — codec is what turns your object into one and back — so caching a parsed JSON object, an extracted record, or a tool-call is the same one-embedding round trip as caching text. On a hit the stored text is decoded; on a miss compute runs, its result is encoded and cached, and the same value is returned. See ResponseCodec for the round-trip contract.

val weather: Weather = cache.getOrPut(prompt, weatherCodec) { llm.extractWeather(it) }