Why your weekend project is invisible to ChatGPT
Three gaps every AI-built site ships with, the curl command that shows each one, and roughly twenty minutes of work to close them.
A site built in a weekend usually works. It loads, it looks right, the buttons do what they should. What it often cannot do is explain itself to a machine that arrives without a browser, without patience, and without any interest in your animations.
That machine is what decides whether an answer engine can cite you. Here are the three gaps that show up most often, in the order they cost you.
1. The page is empty until JavaScript runs
Most AI crawlers fetch your HTML and read it. They do not wait for a framework
to boot, hydrate and fill in the content. If your page ships an empty <div id="root">
and paints everything client-side, what the crawler sees is an empty div.
Googlebot will render JavaScript, eventually, on a second pass it schedules when it feels like it. Most answer-engine crawlers will not.
The fix: server-render or pre-render the pages that matter. In practice this
means static generation for anything that is not user-specific. Check what you
actually ship with curl -s https://yourdomain.com | head -100, if your
headline is not in there, neither is it in the index.
2. There is no metadata to lift
An answer engine quoting you needs a title, a description, and enough structure to know which part of the page answers the question. Site builders and scaffolding tools generate a placeholder title and an empty description, and almost nobody goes back to fill them in.
An empty <meta name="description"> is worse than a missing one. A missing tag
lets the engine compose something from your content. An empty one is an explicit
statement that there is nothing to say.
The fix: a real title and description per page, headings in order, and
Article or FAQPage structured data where it genuinely applies. Not on every
page: schema that lies about what a page contains is its own problem.
3. There is no llms.txt
llms.txt is a plain-text file at your site root that tells AI crawlers what
your site is and which pages are worth reading. It is young, it is not universally
honoured, and it takes about ten minutes to write.
It is not a ranking trick. It is closer to a README for a machine: this is the product, these are the pages that explain it, here is what is a marketing page and here is what is documentation.
The fix: write one. Keep it short and honest: a summary paragraph and a linked list of your genuinely useful pages beats an exhaustive dump of every URL.
What this adds up to
None of these three is hard. All three are invisible, which is why they survive launch: nothing in your browser tells you that the page a crawler receives is different from the page you see.
That gap between what you see and what a crawler gets is the entire problem, and it is what a scan is for. Crawld’s free scorecard fetches your pages the way those crawlers do and reports what came back: including, explicitly, the checks it could not run. A check that could not run is never counted as a pass.