Newton’s Laser Tag went live on a Tuesday evening. Four seconds later CI went red, and it stayed red for three days.
Nothing was broken.
CI rebuilds the whole site every time I change anything, runs every check I’ve written against the result, and refuses to publish if one of them fails. Red means it stopped. For three days it was red, and it was right to be.
This site lists companies Actium builds and backs, and some of them haven’t launched yet. Naming one early is a mistake with no undo (the page is cached and scraped and sitting in somebody’s feed before you spot it), so every build ran a check that searched the finished pages for the names of unlaunched companies and refused to publish if it found one. That worked for two days. Then a company launched, its name legitimately entered the site, and the check did exactly what it had been written to do.
57 consecutive failures. Zero successes in between. The commit message I wrote when I finally fixed it says 60; I’d guessed at a number I could have counted.
Nobody noticed
CI here runs as two steps in order: build the site, then deploy it. The second one waits on the first, which takes two lines to say:
deploy:
needs: buildThat reads: don’t deploy until the build has finished and passed. It’s the right rule. You don’t want to ship a site that failed its own checks.
Now watch what that rule does when the build fails. Deploy doesn’t fail with it. Deploy is skipped, and GitHub paints skipped jobs gray. Red is an alarm. Gray looks like a step that simply had nothing to do that day.
The site looked fine too, because Cloudflare kept serving the last good deploy,
which was recent and correct, so nothing 404’d and nothing looked stale and no
monitor fired. For three days every change I pushed landed on main and went
absolutely nowhere, and the only signal was a red check on a page you don’t
think to visit when the site plainly works.
The defect isn’t what it looks like
I wanted to write this up as “someone forgot to update the list.” That’s wrong, and if you believe it you’ll build the same bug again with a sticky note attached.
The check was a list of company names typed out by hand. Read what that construction actually requires: the list has to be edited by the same action it exists to guard. Launching is the moment the policy changes, and it’s also the moment you’re doing twelve other things.
And it fails backwards. The check doesn’t break when you make a mistake. It breaks when you succeed: every correct launch, forever, until someone remembers to edit a YAML file.
Deriving it instead
Every company on this site has a file, and each file opens with a small block of
settings: name, URL, and a draft flag saying whether it’s public yet. The
replacement check reads those blocks, keeps the companies still marked
draft: true, and assembles its forbidden list from what it finds: the company
name, the name with punctuation stripped, the domain from its URL, and the
distinctive part of that domain.
There’s nothing left to maintain. Flipping one company’s flag to draft: false
is what publishes its page, and that same flip is what drops its name from the
forbidden list. The gate isn’t a copy of the policy anymore. It’s a function of
it.
Two details mattered more than I expected. The first search was case-insensitive and matched anywhere inside a word, which is a problem the moment a company name happens to sit inside an ordinary English word: the check fires on innocent prose and you learn to ignore it, which is the worst thing a check can teach you. Terms have to match as whole words now.
The second is that matching file contents was never enough. Every built file’s path gets checked too, whatever its type, because a filename can carry a name even when nothing inside the file does. Only text files get read for content, and that asymmetry is deliberate: built filenames also carry a random string of characters so browsers know when to re-download them, and every so often that random string spells something.
It runs inside npm run build too, instead of living in the workflow file. The
old one could only be discovered by pushing and waiting. Fifty-seven times, as
it turned out.
The part I can’t show you
The obvious thing to put here is the original grep line, so you can see the shape of it.
I can’t. It names four companies that still haven’t launched, so pasting it would put all four into the public build, and the current gate would catch it and fail this page.
Which is funnier than it is instructive. But it’s the same property either way: the rule comes from the current state of the content, so it covers this sentence, written after the fact, without anyone having to remember that it should.