A learning project — built with Claude Code
An agentic system that generates marketing campaigns the way code factories generate software — and measures them honestly enough to be told it was wrong.
Follow along
I am building this in the open. When there is a real result — including the ones that say the idea did not work — I will send it to you.
No more than one email a week. Unsubscribe anytime.
Marketing experiments usually fail for unglamorous reasons: the thing being measured is a proxy for the thing that matters, several changes ship at once, or a promising arm gets switched off before it could possibly have converted. This system is an attempt to make those failures structurally hard rather than merely discouraged.
A hypothesis and a metric contract are written down before anything is built. The contract is immutable for the run.
Strategy, copy, design and landing-page agents produce a control plus one-factor-at-a-time variants.
A critic runs blocking checks — policy, asset bounds, tracking self-test, load budget. A human approves the spend.
Traffic is split behind one stable URL, rewards mature, a bandit reallocates, and the run ends in a verdict.
The interesting part is not the bandit. It is that the system runs at three different tempos at once, and each one is only allowed to touch what it can be held responsible for.
Each of these exists because the alternative produces a number that looks like evidence and is not.
The reward can only ever be the conversion the campaign exists to move. Clicks, CTR, sessions and form-starts are structurally excluded — not discouraged, rejected by code. Every one of them correlates with the thing you want and none of them is the thing you want.
A control plus one-factor-at-a-time treatments. Change the headline, the proof style, the form friction and the layout at once and a win tells you nothing about why.
The variant never appears in the URL the ad platform sees. Swapping final URLs to test pages would hand the experiment to someone else's optimiser.
Arms keep a minimum share of traffic until they have enough sessions and at least one full conversion cycle. Bandits that collapse early are confidently wrong.
The critic's checks and the human spend approval are refusals, not warnings. An approval records who authorised what, against which build, for how much.
We tried pointing the factory at Amazon affiliate products. The conversion happens on Amazon, where nothing can be instrumented — but the outbound click is right there, instant and easy. It is also worthless: a page that wins clicks from badly matched visitors earns nothing. The fix was to make the click a diagnostic and put per-variant attribution on a different rail. Every measurement system has this shape of temptation.
The same affiliate idea died on a spreadsheet, not on a page. A $35 product at 3% commission returns about two cents per session against clicks that cost thirty to two hundred. No amount of landing-page craft closes a 20x gap. The planner now refuses to stay quiet about it: it computes sessions-per-cell for your minimum detectable effect and says when a test cannot finish inside its own duration cap.
If a conversion arrives days after the click, a sampler that updates nightly punishes whichever arm was most recently shown. Only matured cohorts feed the posterior, and evidence is discounted with a half-life so the system tracks a moving world without thrashing.
Holding type, spacing and colour constant across variants is not an aesthetic preference. It is what makes the measured difference attributable to the thing under test instead of to a bundle of unrelated changes.
The most expensive failure in this build had nothing to do with bandits. A host's API capped uploads at 8KB per field, so source had to be shipped as hand-copied base64 chunks; two chunks from different builds got mixed and the whole brain crash-looped on 'invalid compressed data'. The fix was a checksum that names the failure, and then removing the hand-copying entirely. Verify the boring parts.
Approval writes the hypothesis, the plan, the build hashes and the authorised budget to a file that gets committed. Afterwards it is checkable: this is what we said we would test, before the data arrived.
Per-session variant assignment behind a stable URL, propensity logged on every assignment, allocation read from Edge Config with the bundled snapshot as fallback.
An always-on ingestion endpoint, and a daily job that matures rewards, updates the bandit and republishes the allocation.
Write the hypothesis and product brief; the system plans the experiment, generates the pages, runs the blocking gate, and records your approval as the spend authorisation.
A case study that only lists successes is a brochure. These are the honest gaps as of this writing:
The pages this site serves at other paths are generated by the system described above, and this page is composed from the same design system those pages use.
Follow along
I am building this in the open. When there is a real result — including the ones that say the idea did not work — I will send it to you.
No more than one email a week. Unsubscribe anytime.