getOrPutStreaming

A getOrPut for streaming completions: forwards a streamed answer to the caller as it arrives, keeps it, and replays it chunk for chunk on a later hit.

Every chat interface in production streams, because a first token at 300 ms is the difference between a product that feels alive and one that does not. Without this a streaming team either cannot put a cache on that path at all, or has to buffer the whole response before showing any of it, which spends the latency they were streaming for and makes every miss slower than no cache.

Two rules make it safe rather than merely useful, and neither is configurable.

The lookup, the guards and the verifier all run before the first token is served, never during. A token already handed to the caller cannot be taken back, so a guard that fired halfway through would have served a wrong answer to somebody who is already reading it. Everything that decides whether to serve happens while this function suspends; collection only starts once the decision is made.

And a stream that fails partway is not written. The entry is stored only when the upstream flow completes normally, so a truncated answer never becomes a cache entry. That would be worse than no entry at all, because it would be served confidently to everyone who asked that question afterwards and would look exactly like a complete one. Cancellation counts as failure here for the same reason.

On a hit the answer is replayed according to replay, which defaults to the original chunk boundaries with no delay. See StreamReplay, including why no option reproduces the original timing. An entry written by any other path has no boundaries and replays whole.

On a miss the flow from compute is passed straight through, chunk by chunk, while being accumulated. Empty chunks are forwarded but not recorded: they carry nothing to replay.

Concurrent misses are coalesced, on the same coalesceConcurrentMisses switch as getOrPut and for the same reason: fifty requests for one new prompt arriving together is the case a cache exists for, and streaming is the path a chat product actually serves users on. The first collector opens the provider stream; the rest attach to it, are replayed whatever has already arrived, and then follow it live. There is one provider call and everyone sees the same chunks.

Three consequences worth stating. A failure reaches every attached collector and writes nothing, because a truncated answer served confidently to fifty people is fifty wrong answers rather than one. The provider stream is stopped when the last collector leaves rather than the first, so a caller who closes a tab no longer takes the answer away from everyone else, while a lone caller who cancels still cancels the stream and still caches nothing. And an attached caller sees the provider's own chunk boundaries whatever replay it asked for, because there is one live stream and replay describes how a stored answer is cut up.

The returned flow is cold: nothing is computed or streamed until it is collected, though the embedding and the lookup are already done. Write-behind does not apply to this path. If the embedder throws under EmbedFailurePolicy.FALL_BACK_TO_COMPUTE, the raw stream from compute is returned uncached.

cache.getOrPutStreaming(prompt) { llm.completeStreaming(it) }.collect { chunk -> print(chunk) }

The getOrPutStreaming that chooses how a hit is replayed.

A separate overload rather than a parameter on the one above, and replay carries no default, for the same two reasons: inserting anything ahead of a trailing lambda rebinds it for every caller passing scope or metadata positionally, which is a source break 2.x does not take; and a default here would make getOrPutStreaming(prompt) { … } ambiguous between the two.