Before a system can skip work it has already done, it has to answer one question, which is what does this change affect? It sounds like a small piece of bookkeeping until you try to answer it precisely, at which point it turns into the whole problem, because the cost of getting it wrong is not a slower site but a wrong one.
We are in the middle of this work right now on the Reeweb publisher, with roughly half of it running and the other half still on paper, and I want to write about it from here rather than from the comfortable position afterwards where everything looks like it was obvious. Notes written after the fact tend to leave out the part where you did not know, and the part where you did not know is usually the part worth reading.
The Same Question, Asked at Different Times
In an earlier note I argued that rendering before publication and rendering when a request arrives are different architectures, and I still think that is right, but working on this has shown me how much the two models have in common underneath. Both of them are trying to avoid rendering everything, both of them therefore need to know which outputs a change invalidates, and neither of them can be correct without an answer to that question.
What differs is when you are forced to answer it. A request-time system answers under the worst possible conditions, since a visitor is waiting, the answer has to be produced in milliseconds, and the machinery for it has to be general enough to work for any application built on the platform, which is why explicit tagging and manual path invalidation exist and why those APIs put the burden on the application developer to declare what depends on what. A system that regenerates before publication answers under the best possible conditions, since nobody is waiting, seconds are acceptable, and if the answer is unclear there is always the option of doing the work anyway.
That last freedom is the entire advantage, and it is worth being precise about why. When you have to answer during a request, being wrong in the safe direction means rendering more than necessary while somebody waits, so the pressure is toward answering quickly and narrowly. When you answer before publication, being wrong in the safe direction costs a few seconds of a machine's time that nobody experiences, so the pressure is toward answering conservatively. Same question, opposite incentives, and the incentive is what determines how the system behaves when it is uncertain.
What Already Runs
The publisher exists and does the coarse version of this, which is to say it treats any change as affecting everything, regenerates the whole site, and shows a person exactly what came out different. That is not sophisticated but it is correct, and correct was the right thing to build first.
It watches the repository, and when the head commit changes on the publish branch it schedules a render after a short quiet period, so that a burst of commits collapses into one pass instead of starting a generation for every push. When the generation finishes, the interesting part happens, which is that the new dist/ is compared against .deployed/, a directory holding exactly what was last put in front of the public:
const old_file = old_files.get(path);
const new_file = new_files.get(path);
if (old_file?.hash === new_file?.hash) continue;
const kind = !old_file ? "added" : !new_file ? "deleted" : "changed";
Every file on both sides is hashed with SHA-256 and compared by content, so the diff answers a question that matters more than it first appears: not what did the generator write, which is everything, but what would actually change for a visitor, which is usually a handful of files. A person looks at that list and decides whether to publish, and the deploy is a separate deliberate act rather than a consequence of committing.
Two details in there took longer to get right than they look. The snapshot of .deployed/ is only taken after a deployment succeeds, which is what makes the diff mean "difference from what the public currently has" rather than "difference from the last time I rendered." And a failed generation restores dist/ from that snapshot, so a broken render never leaves a half-written edition sitting on disk where the preview server can serve it.
This is a full regeneration every time, and for a site of a few hundred pages that finishes in seconds, so there has been no reason to make it cleverer. The reason we are making it cleverer anyway is that the same publisher is meant to serve projects where content comes from an editorial team rather than from commits, and where waiting a minute to see a correction go live is the difference between a tool people use and a tool people work around.
Where It Gets Hard
The moment you want to regenerate only part of a site, you need to answer the question in the title with enough confidence to not do work, and this is where I found the problem to be considerably deeper than I expected when we started sketching it.
A page does not depend only on its own file. It depends on its layout, on any includes and components that layout pulls in, on the translation files for its locale, on shared page data and helpers, on locale configuration, and on whatever a sibling data loader happens to fetch, which that loader does not declare anywhere. Change a shared layout and you have changed every page. Change one translation key and you have changed every page that renders it, which nothing currently tracks. Change a route name translation and you have not merely changed page text but moved the output path, which means links elsewhere on the site now point at the old location and the old file has to be deleted.
The fan-out runs the other way too. A single Markdown article contributes to its own page, to a sidebar group, to a paginated listing, to the RSS and JSON feeds, to the sitemap, to an Open Graph image, and to one or more search indexes, so regenerating "the page" while leaving the aggregates stale produces a site that is internally inconsistent in ways a visitor notices before any monitoring does. Aggregate outputs are the awkward case, because the ownership runs backwards: a rendered page belongs to the source file that produced it, while a search index belongs to no single page and to all of them at once, which means the neat mapping from one input to its outputs, the mapping the whole optimization depends on, simply does not exist for the artifacts that matter most to a reader trying to find something. Updating one article means rebuilding every index that contains it, and the index does not get smaller as the change gets smaller.
Then there is deletion, which is the part that surprised me most. Writing a changed page is easy. Knowing which files should no longer exist is not, and it needs a record of which outputs belong to which unit of work, because a renamed route leaves an orphan, a shortened pagination leaves numbered pages that should be gone, and a changed script bundle leaves a content-hashed file nothing references any more. A full regeneration handles all of this by clearing the directory first, which is a crude solution and also a complete one.
Under-invalidation is the real risk. A fast regeneration that leaves stale pages behind is worse than a slow one that does not.
That sentence is the design position, and everything else follows from it. A site that is slightly slow to publish is an inconvenience, while a site that quietly serves a page that should have changed is a correctness failure, and the second is worse because nobody finds out until a person notices something wrong and by then nobody trusts the publishing system.
The Position We Have Taken
Since being wrong in one direction is much more expensive than being wrong in the other, the design does not try to be clever about ambiguity. It tries to recognize ambiguity and then decline to be clever.
Concretely, that means the publisher keeps a record of what depends on what and which outputs belong to which unit, checks that record against the current state before trusting it, and takes the selective path only when the affected set can be established. When anything is unclear, when the record is missing or does not match the current commit, when a change touches a file type outside the narrow set we understand, when a route might have moved, or when a data loader's dependencies cannot be determined, it regenerates everything. Full regeneration is not the failure mode, it is the default, and the selective path is an optimization that has to prove it applies.
The test we set for ourselves is that an incremental pass must produce byte-for-byte identical output to a complete regeneration of the same commit, with no file differing and no file left behind. That is a demanding standard and it is deliberately the only one we will accept, because any weaker test would let exactly the under-invalidation bug through, and a bug of that shape would not announce itself.
The way it gets built follows from the same caution, which is one dependency class at a time, each behind its own switch, with parity checks running against full regenerations before that class is trusted. It is slower than doing it all at once and it means the feature arrives in pieces rather than as an announcement, but a publishing system is a thing people have to trust, and trust is not something you can add in a later release.
Why Write This Now
There is an obvious argument for waiting, which is that a finished feature makes a better post than an unfinished one, and that writing about a design before it is built risks describing something that turns out differently.
I think that risk is worth taking, because what is actually useful in a note like this is not the outcome but the reasoning, and the reasoning is fully formed already: the question is what a change affects, the danger is answering it too confidently, and the response is to make full regeneration the default and require proof before skipping work. If the implementation teaches us something that changes that reasoning, then that is the more interesting post, and it can only be written if the current thinking is on the record first.
There is also a smaller reason, which is that our own planning documents for this work drove us to be precise about vocabulary in a way that the earlier note came out of, since you cannot design an incremental generator without first deciding what incremental means, and that turned out to be a better prompt for thinking clearly than any amount of discussion in the abstract. Building the thing is what forces the definitions to be honest.
So this is where we are: the coarse version runs and is trustworthy, the selective version is designed and not yet built, and the part we are least certain about is how much of the dependency graph can be established honestly rather than guessed at. When we know, I will write that down too, including the parts where this turned out to be wrong.