TokenPrices

What one model call in a scope costs, declared by the caller.

The README does arithmetic that turns a hit rate into money, by hand, in prose, with numbers the reader has to supply from their own invoice. That is the calculation anybody deciding whether to adopt a cache actually cares about, and until this the library could not do it: CacheStats counted lookups, hits, misses and rejections, and none of that is money. A hit on a two hundred token answer from a cheap model and a hit on a four thousand token answer from an expensive one were one increment each.

The price comes from you, not from here. kmemo ships no table of provider prices, for the same reason it ships no embedding model: prices change weekly, a vendored price list is wrong the month after it ships, and a library that quietly reports the wrong saving is worse than one that reports none. You state the number, the cache multiplies, and Savings carries the number back out with the figure so nobody reads the total without its assumption.

The token counts come from your metadata. CacheEntry.metadata is free-form caller data returned untouched on a hit, and token counts are what people put in it. The cache reads them by key from the entry that was served, so the saving is the cost of the call that was actually avoided rather than an average applied to a hit count.

val cache = semanticCache(embedder) {
    prices["gpt-4o"] = TokenPrices(
        currency = "USD",
        perInputToken = 2.50 / 1_000_000,
        perOutputToken = 10.00 / 1_000_000,
    )
}
cache.getOrPut(prompt, scope = "gpt-4o", metadata = mapOf(
    "inputTokens" to usage.input.toString(),
    "outputTokens" to usage.output.toString(),
)) { llm.complete(it) }

cache.stats().savings["gpt-4o"]   // Savings(amount=..., currency=USD, hits=..., ...)

Parameters