Design · Orchestration
Semantic Caching
Embed each incoming query; if a semantically similar past query is in the cache, return its stored answer instead of calling the model. Cache-hit queries cost nothing.
Verified against github.com
How it works
Embed each incoming query; if a semantically similar past query is in the cache, return its stored answer instead of calling the model. Cache-hit queries cost nothing.
At a glance
TechniqueCaching
Cost leverLower, cache hits skip the model entirely
Latency leverMuch lower on cache hits (ms vs seconds)
Quality leverDepends on similarity threshold, too loose returns stale answers
ExampleGPTCache (Zilliz) · Portkey semantic cache
When to use
Workloads with repetitive or near-duplicate queries (FAQ bots, support, docs search).
Basis
Verified against
https://github.com/zilliztech/GPTCache The corpus does not rewrite vendor documentation. Where copy is quoted verbatim it is marked as such and attributed to its source.