CachingChatModel

A LangChain4j ChatModel that puts a SemanticCache in front of another model.

LangChain4j ships no semantic cache, so this is the drop-in: wrap the model you already have, and a repeated-in-meaning request is served from the cache — with kmemo's guards stopping the answer for 250 USD from being served to a request about 100.

val model: ChatModel = CachingChatModel(openAiChatModel, cache)
val answer = model.chat("What is the capital of France?")

The cache key is the whole conversation — every message, in order — not just the last one, so a question asked after a different exchange is a different key and cannot be served the earlier answer. A request whose messages are not plain text (a tool result, a multi-part user message) is passed straight through, uncached, rather than keyed on something lossy. Anything that changes what a correct answer looks like — the model, temperature, the system prompt if it is not already in the messages — belongs in the scope.

On a hit the wrapped model is not called; the cached text comes back as an AiMessage. On a miss the delegate runs and its assistant reply is cached. The cache is a suspend API and ChatModel is blocking, so lookups run inside runBlocking on the calling thread — which on a hit skips the model call entirely, the whole point.

Parameters