5 silent production failures and how to avoid them
Imagine your app is running. People are using it. And then something breaks — but you don’t hear a bang. The screen doesn’t flash red. The page simply won’t load, someone can’t log in, and at the end of the month you get a bill you didn’t expect.
The most dangerous failures are silent. They don’t shout. And that’s exactly why they can run quietly for days before anyone notices. Here are five such situations — described so it’s clear what to ask the contractor or the AI before you go live with real users.
1. One bad click takes down the app for everyone
Imagine a jam in one lane that stops the whole city. That’s how a poorly protected app behaves: one user does something unusual — types a strange character, logs in with a passwordless account — and the whole service shuts down. Not just for them. For everyone at once.
That’s dangerous because you can’t predict it. All it takes is one unexpected case nobody checked.
What to take care of: ask that an error in one request breaks only that one request, not the whole app (this is called a “global error catcher”).
2. No limit — an open door for anyone
Imagine an unstaffed buffet: one person can grab every plate and no one stops them. Same with an app with no limits — one user (or bot) can send thousands of requests at once and overload the system so it stops responding to everyone else.
That’s dangerous because it doesn’t take an attack — sometimes a badly written script on the other side is enough.
What to take care of: ask for a limit on the number of requests per person in a given time (technically: “rate-limit”) — for example “at most 100 requests per minute from one account” — so no single party can clog the whole thing.
3. Someone else’s service went down and dragged you with it
Your app almost always relies on other people’s services — maps, payments, sending email. It’s like building a shop whose electricity is supplied by your neighbour. When the neighbour’s power goes out, your shop stops too. And other people’s services can be unreliable: they slow down, block, sometimes return junk instead of data.
That’s dangerous because your app can hang forever, waiting for a reply that never comes.
What to take care of: ask that every connection to an outside service has a time limit (a “timeout”) and an automatic retry — so a short failure at the neighbour’s doesn’t hang everything at yours.
4. A loop that quietly drains the budget
Some services charge money for every use — for example an AI that answers questions, or sending email. Imagine a tap that drips all night: a single drop costs nothing, but by morning your flat is flooded.
If something in the app loops and calls a paid service endlessly — or someone deliberately abuses it — the bill grows quietly. You find out only from the invoice.
What to take care of: ask for a hard cost limit per user (e.g. monthly), with a clear message “you’ve used your limit, come back in X days”.
5. “I sent the email” doesn’t mean it arrived
You click “send”, the system says “sent” — and you’re at ease. But the email landed in spam or nowhere. It’s like dropping a letter into a mailbox that turned out to be bricked up.
That’s dangerous because the problem is invisible. Everything looks like it’s working, and customers simply don’t get confirmations, invoices or welcome messages.
What to take care of: ask for a real deliverability test — that is, a check that the email actually reaches the inbox, not just “left” the server. In practice this starts with proper sender authentication (SPF, DKIM and DMARC records), without which mail often lands in spam.
In short
- Silent failures are the most dangerous — they don’t shout, they quietly hang, log people out or drain the budget.
- Assume the world is unreliable: other people’s services go down, users do unforeseen things, and paid tools charge for every click.
- Before going live with real users, ask the contractor or the AI for four things: a global error catcher, limits (of time and cost), retries for outside services, and a real email deliverability test.