The Most Expensive Bugs Are Usually Not the Hardest Ones

The bug I remember most was not a concurrency issue or a memory leak. It was a payment assumption: every failed transaction was safe to retry. Under one timing condition, a retry charged the customer twice.

The code was small. The impact was not.

Expensive bugs live at boundaries: between frontend and backend, software and humans, or “failed” and “not finished”. Unit tests can stay green when the tests share the same assumption as the implementation.

During an incident, ask which states are possible, who owns the final decision, whether an operation is safe to repeat, and what the user sees when the system is uncertain. Model pending, succeeded, failed, and unknown explicitly instead of forcing everything into a boolean.

Logs should include a correlation ID, operation, previous and next state, and decision reason—never secrets. Metrics should describe impact: duplicate charges, retry rate, and recovery time.

A good fix adds a regression test, documents the assumption, and creates an alert before users discover the same failure. Difficult bugs are not automatically expensive. A simple bug touching money, trust, or personal data usually is.

---

Business bugs are more dangerous than algorithm bugs

Algorithm bugs usually have inputs and outputs that can be compared. Business bugs are quieter: a discount is applied twice, a deleted account still receives email, or a monthly report uses the wrong timezone. The system runs, but the decision is wrong.

To find these bugs, I write scenarios from the user’s perspective. “A customer presses pay twice while the network is slow” is more useful than a test called should_process_payment. The scenario forces us to consider idempotency keys, timeouts, retries, and the UI state.

Test time boundaries as well: midnight, month-end, delayed jobs, and daylight-saving changes. Use a controllable clock instead of making a test depend on whatever Date.now() returns.

After an incident, avoid saying that someone was not careful enough. Look for the system condition that made the mistake easy: ambiguous states, an API that accepts retries without idempotency, or a dashboard that hides uncertainty. The best fix combines code, a regression test, a metric, and a runbook.

We cannot remove every bug. We can make the same bug harder to create and faster to detect.

Sources

• https://sre.google/sre-book/monitoring-distributed-systems/
• https://martinfowler.com/articles/practical-test-pyramid.html
• https://owasp.org/www-project-top-ten/