Access — Can a Crawler Read Your Page at All?
If a machine can’t read your page, it can’t quote you.
Access is the first pillar of the AEO Canon and a binary gate — if an AI crawler can't read your page, it can't quote you. Most AI crawlers execute no JavaScript, so server-rendered, fast, allowed HTML is the price of admission.
Access is the first pillar of the AEO Canon, and it is a binary gate: if an AI crawler can't read your page, it can't quote you. There is no partial credit and no workaround — every other pillar is irrelevant until this one passes. The good news is that access problems are specific and fixable.
The AEO Canon · the cascade
Pillar 1 · Access — If a machine can’t read your page, it can’t quote you.
Why is Access a binary gate?
Access is a binary gate because an answer engine can only cite text it has actually read. If your content isn't in the HTML the crawler receives, you are not a weak candidate for citation — you are not a candidate at all. This is what makes Access different from every other pillar: Authority, Credibility, and the rest are matters of degree, but Access is pass/fail. A brilliantly written, perfectly authoritative page behind a blocked crawler earns exactly nothing.
That's why the Canon puts Access first and treats it as a cascade gate: there is no point optimizing anything downstream until you've confirmed the machine can read you. Most "Why isn't my site cited by AI?" investigations end here.
Do AI crawlers actually read JavaScript?
No — most AI crawlers read raw HTML and execute essentially no JavaScript. This is the single most important technical fact in AEO, and it's measured, not theoretical.
Vercel, with MERJ, analyzed more than 500 million GPTBot requests and found zero evidence of JavaScript execution. The crawler fetches your raw HTML and stops; anything assembled client-side — content injected by React, Vue, or any framework after load — is simply invisible to it. Google's Gemini is the notable exception, because it rides Google's existing rendering pipeline, but for ChatGPT, Claude, Perplexity, and the rest, the rule is: if it's not in the server response, it doesn't exist.
This catches a surprising number of modern sites. A single-page app that looks
perfect in a browser can serve an AI crawler an empty shell — a <div id="root">
and a script tag. The content humans see is rendered by JavaScript the crawler
never runs.
Why does speed belong in Access?
Speed belongs in Access because a crawler that times out, gets throttled, or abandons a slow page never reads your content — so slowness is a soft version of being unreadable. SE Ranking found that pages with a First Contentful Paint under 0.4 seconds earned roughly three times more citations than slower pages. Whether that's pure causation or fast pages simply correlating with well-built sites, the practical conclusion is the same: a fast, lightweight, server-rendered page is both more crawlable and more favored. Core Web Vitals were always good practice; for AEO they're table stakes.
How do you apply the Access pillar?
Apply Access by making your content exist in the raw HTML, allowing the crawlers, and keeping the page fast — then verifying it. Work the checklist below; if you can't tick every box, this is where your AEO program starts.
- 1
Serve content server-side
Use server-side rendering or static generation so your content is in the initial HTML response, not assembled by client-side JavaScript.
- 2
Allow AI crawlers in robots.txt
Explicitly permit GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and others. A disallow here is an invisible AEO killer.
- 3
Stay fast
Optimize Core Web Vitals — aim for a very fast First Contentful Paint. Speed aids both crawling and apparent favorability.
- 4
Verify as a crawler
Fetch your page with JavaScript disabled (or view the raw HTML). If your main content isn't there, the crawler doesn't see it either.
Apply the Access pillar
0 / 5
Each unchecked box is a place a competitor can beat you to the AI answer.
What are the most common Access mistakes?
The most common Access mistakes are invisible because the site looks fine to humans — the failure only shows up when a JavaScript-free crawler arrives.
The silent killers of Access
Client-side rendering: a single-page app that serves crawlers an empty shell. Blocking AI crawlers: a robots.txt disallow, or aggressive bot filtering at the CDN, that quietly excludes you. Content behind interaction: answers hidden in tabs, accordions, or modals that only populate on click. Slowness: pages so heavy the crawler times out. Each makes great content uncitable — and none of them are visible in a normal browser test.
A particularly costly version: blocking AI crawlers "to protect content," then wondering why AI never recommends you. You can't be cited by a system you've shut the door on. If your goal is visibility, the door has to be open.
Where Access fits in the Canon
Access is the foundation beneath the foundation — pass it, and you've earned the right to compete on the pillars that follow. It answers the first question of the Foundation layer: can the machine use you at all? Once a crawler can read you, the next question is whether you're answering what people actually ask — Alignment — and then whether your answer is easy to lift, Extractability.
If you suspect Access is your problem, the fastest diagnosis is in why isn't my site being cited by AI?, and the full framework is The AEO Canon. Fix the door first; everything else depends on it.
Frequently asked questions
- Why can't AI cite my website?
- The most common reason is access — AI crawlers can't read your page. Most fetch raw HTML and execute essentially no JavaScript, so client-rendered content is invisible to them. If your robots.txt blocks AI crawlers, or your content only appears after JavaScript runs, you're absent from the candidate set before any other factor matters.
- Do AI crawlers run JavaScript?
- Mostly no. Vercel and MERJ, analyzing over 500 million GPTBot requests, found zero evidence of JavaScript execution — the crawler reads the raw HTML and nothing more. Google's Gemini is the exception, since it uses Google's rendering infrastructure. For everything else, assume your content must exist in the server-rendered HTML.
- How do I let AI crawlers access my site?
- Allow them in robots.txt (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and others), serve server-side-rendered or static HTML so content exists without JavaScript, and keep the page fast. Then verify by fetching your page as a crawler would — with JavaScript disabled — and confirming the content is there.
- Does page speed affect AI citations?
- Yes. Fast pages are crawled more reliably and appear to be favored — SE Ranking found pages with a First Contentful Paint under 0.4 seconds earned roughly three times more citations. Speed is part of access because a crawler that times out or is throttled never sees your content.
Last updated .