Newton’s Laser Tag went live on a Tuesday evening. Four seconds later CI went red, and it stayed red for three days.
CI rebuilds the whole site whenever I change anything. It runs my checks against the result, and it refuses to publish if one fails. Red means it stopped, and by its own rule it was right to stop.
This site lists the companies Actium builds and backs, and some of them haven’t launched yet. Naming one early is a mistake you can’t undo: the page is cached, scraped and in somebody’s feed before you spot it. So every build searched the finished pages for the names of unlaunched companies. If it found one, it refused to publish. That worked for two days. Then a company launched, its name rightly appeared on the site, and the check did what it was written to do.
The check failed 57 builds in a row. The commit message I wrote when I finally fixed it says 60; I’d guessed at a number I could have counted.
Why I didn’t notice
CI here runs as two steps in order: build the site, then deploy it. The second one waits on the first, which takes two lines to say:
deploy:
needs: buildThat reads: don’t deploy until the build has finished and passed. It’s the right rule. You don’t want to ship a site that failed its own checks.
But when the build fails, deploy doesn’t fail with it. Deploy is skipped, and GitHub paints skipped jobs gray. Red is an alarm. Gray looks like a step that had nothing to do that day.
The site looked fine too. Cloudflare kept serving the last good deploy, which
was recent and correct. No page 404’d or looked stale, and no monitor fired. For
three days the changes I pushed to main didn’t deploy. The failure showed only
as a red check, on a page you don’t think to visit while the site works.
Why the check failed
I wanted to write this up as “someone forgot to update the list.” That’s wrong. If you believe it, you’ll build the same bug again and add a reminder.
The check was a list of company names typed out by hand. That list had to change during each launch, the event it existed to guard. Launching is the moment the policy changes, and it’s also the moment you’re doing twelve other things. So every correct launch would break the check, until someone remembered to edit a YAML file.
Deriving the list instead
The site keeps one file per company. Each file opens with a small block of
settings: name, URL, and a draft flag that says whether it’s public yet.
The new check reads those blocks and keeps the companies still marked
draft: true. Then it builds its forbidden list from what it finds:
- The company name
- The name with punctuation stripped
- The domain from its URL
- The distinctive part of that domain
There’s no hand-typed list left to maintain. Flipping a company’s flag to
draft: false publishes its page. The same flip drops its name from the
forbidden list.
Two details mattered more than I expected. The old search ignored case and matched inside words, so a company name inside an ordinary English word would fail a build on innocent prose. Now terms have to match as whole words.
The second detail is that searching file contents wasn’t enough. Each built file’s path gets checked too, whatever its type, because a filename can carry a name even when the file’s contents don’t. Only text files get read for content, because scanning an image’s bytes throws false alarms when a run of them happens to spell a word.
The check runs inside npm run build now, instead of as a step in the workflow
file. With the old one, I could only see a failure by pushing and waiting. I did
that 57 times.
The line I can’t show you
The obvious thing to show here is the original grep line. I can’t. It names companies that still haven’t launched. Pasting it would put their names into the public build, and the current check would catch it and fail this page.
That’s funnier than it is useful. The check builds its list from the content as it is now. So it covers this sentence too, written after the fix, and I didn’t have to remember that it should.