Automated Rollback Is Not AI. It Is Competent Defaults.

Part one of a four part series on building technology infrastructure that survives success, following my NITHUB Innovation Fair masterclass, Building Technology Infrastructure for Scale. This part is about the unglamorous thing that has to be in place before you go viral, not after.

Picture the good version of your worst day. The thing you built is working. A link is going around, signups are climbing, and for the first time real people are depending on your product all at once. Now look at what you are actually doing during that window. You are shipping. A hotfix for the bug three thousand new users just found, a quick change to handle the load, a copy tweak someone asked for on the busiest hour of your life. Growth does not slow your deploys down. It speeds them up, and it does it exactly when a mistake is most expensive.

So here is the failure I want to talk about first, and it is not the traffic. It is you, shipping a broken change into your biggest moment. On stage I show this on purpose: a new route in the API reads an environment variable that nobody added, the container starts, the health check on /healthz returns 500, and the deploy is marked failed. And then nothing else happens. No pager, no dropped traffic, no scramble to a laptop. The previous version is still serving, because the platform never cut over to the broken one. It held the old version, proved the new one was not healthy, and rolled back on its own.

No agent did that. No human did that either. The platform did it, with competent defaults that were in place long before any AI was in the picture. That is the whole point I want to make before I show anyone something clever: the thing that saves you on the day you go viral is not a smart tool watching your back. It is that a failed deploy is a non event by default.

The precondition everyone skips

Most advice about scaling jumps straight to the exciting parts. Autoscaling, load balancers, queues, and now agents. Can it handle the traffic, can it self heal, can the AI find the bug. Good questions, and I spend the rest of this series on the last one. But they all quietly assume something that is usually not true for a young product: that when something goes wrong, and on your big day something will, the system contains the damage on its own.

If a failed deploy takes your site down, nothing you bolt on top of it makes that safer. More traffic just means more people watching it happen. The order of operations matters. Make failure boring first, then add automation on top of it, then, much later, let an agent drive that automation within limits. Skip the first step and everything after it is built on sand.

What competent defaults actually means

Concretely, for a containerized app, it means your deploy is health gated and reversible without a human in the loop.

The new version comes up next to the old one. A health check decides whether it is allowed to take traffic. If it passes, the cutover happens and the old version is stopped. If it fails, the new version is torn down and the old one keeps serving. The deploy goes red, and your users never find out. This is the model my own platform, Reoclo, runs by default, and it is the same idea I have written about before for a single box and for Docker Swarm. The details differ, the property is the same: a deploy cannot hurt you just by failing. You do not need a big team or a big budget to have this. You need it most when you have neither.

The health check is load bearing, so treat it that way

The one place this goes wrong is a health check that lies. If /healthz returns 200 the instant the process starts, before the app has loaded its config, connected to what it needs, and become ready to serve, then the gate is decorative. It will happily cut over to a container that is up but not actually working, which on a high traffic day is its own kind of outage.

So the rule is that the health endpoint has to reflect readiness, not just that the process is alive. In the demo, the app reads a required variable at boot and throws if it is missing. It never reaches the point of answering /healthz at all, so the check fails honestly and the rollback fires. That is the behaviour you want: the app tells the truth about whether it is ready, and the platform acts on it.

Why I lead with this

Later in this series an agent shows up and does something that looks impressive: it reads a failed deploy, finds the missing secret, proposes a fix, and stops to ask permission. A room full of people will want to talk about the AI. I will keep pointing back here, because the only reason I am willing to let an agent near any of this is that the ground underneath it is already safe. The agent operates inside a system that fails closed.

If you are building something you hope will get big, get automated rollback in place before you get the growth, not after. Not because it is clever, it is not, but because it is the difference between a viral moment you remember fondly and one you spend a week apologising for. It is not AI. It is just competent, and competent is the thing you build everything else on.

Next in the series: where an agent actually helps when you are a small team, and where it quietly burns you.