sahansera.dev RSS Feed

no healthy upstream is the kind of error that makes you expect wreckage.

Then you open the dashboards and find… almost nothing. CPU is low. No pods have crashed. The last deployment was hours ago. By the time you refresh the page, the service has recovered by itself.

That was the scene a few weeks ago when I started chasing an intermittent failure in a search ba...

If you have built event-driven systems for any length of time, you have probably internalized the mantra of at-least-once delivery: keep retrying until the work is done, and design everything downstream to be idempotent. It’s good advice. But there’s a subtle failure mode that hides right underneath it - one where your system faithfully reports success, commits its p...