Worth knowing a cache hit rate can be structurally zero and look identical to a broken one.

Of course. On the other hand if you see 8 % gap for similar workloads, averaged across tens of sessions, with the same underlying model, it becomes a pretty clear signal. And I do exclude first request per provider per session from the statistic.

> The context window is very important.

It is everything. My experience with Claude Code is that you have to decide when to compact to make it efficient. It defaults everything to 1M context and it will never keep it in check. It is strange how little cache reads you hit in smol, that may be a configuration issue.

I will look into it more to see if I have configured it wrong but I think the token efficiency hurts cache use as caching only starts at 1024 tokens so for tasks where smol is under or close to 1024 tokens most of them are uncached

There's an env var you can set to tell it when to compact. It's especially helpful for models with big context that lose coherence much earlier.

> how do you properly measure it? > If a problem never manifests, but was prevented, was it really a problem?

For the scheduled half of it, i like inverting the alert. Instead of alerting when a job fails or wait for data leak, have the job check in on every successful run and alert when a check-in doesn't arrive. Dead man's switch. Ten minutes to set up, and it's the difference between "the cron is fine" and a statement that can actually turn out to be false.

Agree that it is difficult to implement for any process, but it is easier to create validation systems than "find the error" processes.

interesting, are you planning to keep the dataset ongoing for over a year?