Four ways my agent lost its provider

On July 24th the concierge on my Mac Mini went dark for the third time in two weeks. No reply on Discord, and the dashboard spelling it out: token refresh failed, invalid_grant, refresh token revoked. Five days earlier the same silence had been a billing problem. Two weeks before that, it had been a process that needed a restart. From the outside these failures are identical: the bot stops answering and the scheduled jobs start failing. Underneath, it hasn’t been the same problem twice.

Kai’s concierge rides one model provider through a subscription, a coupling I wrote down as an accepted risk in week one. Between July 11 and July 26 that risk got exercised four times, each with a different cause and a different fix, and the fix for one applied to another would have made things worse, cost money, or both. Running them down in order is the most useful thing I’ve got from operating an agent system, because none of it showed up while I was building one.

a stale process/

July 11th. The gateway starts throwing 401s, which reads as a bad API key. The key was fine. A stale gateway process was holding dead state, and restarting it cleared everything. The trap is what a 401 trains you to do: rotate credentials, re-authenticate, touch things. The rule that came out of it instead: capture the raw error before restarting anything, because the restart destroys the evidence, and rule out the boring explanation before believing the scary one.

a bounced card/

July 21st. Discord silent, the nine o’clock job failed, and the error said the model provider was rate-limiting requests. The provider’s own app, on my phone, worked fine. The diagnosis took three moves. Reproduce outside Discord, which cleared the gateway and the chat layer as suspects and produced the raw error: HTTP 429, requests per minute, actual over limit, 0/0. You can’t exceed a limit of zero by sending one request. Then try an older model: identical 0/0, so account-wide, not model gating. Then the provider’s usage screen: 31% of the allowance consumed. You can’t be limited to zero by an allowance you’ve spent a third of. Not exhaustion. Suspension. The subscription renewal had bounced off an expired card four days earlier.

The hour that mattered was the one where I nearly bought my way out. The error text blamed the account’s historical API spend, which is developer-platform vocabulary, and for that hour there was a live possibility that a fresh account would start at zero spend, which is tier zero, which is this exact error. I’d been considering a second subscription anyway, to decouple the agent from my personal billing. Buying it mid-diagnosis would have been a coin flip on whether it did anything at all. One usage screen made not-buying safe. The fix was fixing the card.

a revoked token/

July 24th, the one this post opened with. A revoked OAuth refresh token is dead at the provider’s side. A restart can’t help, billing can’t help, and rotating the wrong credential in a panic can hurt. The tell was invalid_grant in the raw error, and the fix was one re-auth command, which the error text named itself. I’ve started treating that as a small test of tooling: an error that tells you its own fix is a vendor that has met this failure before.

an empty tank/

July 26th, Sunday evening, dark again. The auth was fine and the account was fine. The week’s token allowance was gone, plus about five dollars of overflow, resetting Tuesday night. Genuine exhaustion, the failure the other three had been impersonating all month.

My first instinct, again, was a second account. Then I ran a usage report I hadn’t known the gateway shipped. Seven days: 7.5 million tokens. My own conversations with the system were half a percent of that. Two nightly housekeeping jobs were 6.7 million, 89.5 percent, peaking at 2am and 3am, which is those two jobs to the minute. I’d been reasoning about cost for two weeks with no attribution at all.

The mechanism matters more than the number. Actual input and output was 1.16 million tokens. The rest was the same context re-sent on every turn, 586 messages across 21 runs. The jobs weren’t doing more work; they were carrying more with them each turn. That distinction decides everything, because a second account doubles the bucket under the same leak, and you’re back in a fortnight with two dead accounts. More quota fixes exhaustion. It doesn’t fix a loop that grew.

The doctrine that survived the month is now written into the system’s own docs: reproduce the failure outside the chat surface, capture the raw error before any restart, and tell the modes apart before acting, because every wrong guess has a price attached. And one piece of accidental resilience worth naming: the builder agent rides a different provider on a flat rate, so through all four outages the box could still build. The lane with no backup was the one coupled to my personal accounts, which is the call I made in week one and have now paid for four times.

I put five dollars in to bring the concierge back and left both jobs paused. The allowance resets Tuesday night. The jobs won’t be coming back at their old size.