Optimising for Fetch: surviving the single-URL pull

Fetch mode is the single-URL pull. A user pastes a URL, or a model receives one from an earlier step, and the AI tries to retrieve useful content from that exact page. There is no ranking contest here. The URL has already been chosen. Your job is to make the first fetch return facts, links, metadata, schema and a clean summary before the model gives up.

1. What Fetch mode is

Fetch mode is the most literal AI visibility problem. In Search mode, the system decides which page to retrieve. In Fetch mode, the page is known: a pricing page, product detail page, comparison article, support answer, policy, or documentation URL. The failure is not discovery. The failure is readability.

Anthropic's API makes this explicit with web_fetch_20260209, the deployed Combot fetch tool. Claude can fetch URLs explicitly provided by the user, or URLs returned by earlier search/fetch results. OpenAI documents ChatGPT-User for user-triggered page visits, while Perplexity documents Perplexity-User for user actions. The safe engineering assumption is simple: the URL must be useful to a non-browser client.

The practical test is not whether a browser can render the URL after five seconds. It is whether the fetch path returns enough information for a model to answer a narrow question from the URL alone: what this page is, who it is for, what the important terms are, and what the next action should be.

2. The JavaScript trap

The trap is assuming that "works in Chrome" means "works for AI." Anthropic documents that web fetch currently does not support dynamically rendered JavaScript sites. OpenAI and Perplexity document their user-triggered fetchers, but the checked docs do not establish reliable JavaScript execution. Googlebot can render JavaScript in its own indexing pipeline; that is a Googlebot capability, not a guarantee for AI fetchers.

If price, availability, FAQ answers, links, canonical tags or JSON-LD arrive only after hydration, Fetch mode can see an empty shell. Treat the first HTML response as the contract.

3. Lean Render recap

Lean Render is the practical pattern for this problem. It is not old cloaking with a new name. It is a truthful, token-efficient HTML path for agents that need the same facts users see, but do not need the full client application, animation, tracking stack or carousel logic.

The rule is parity. The bot-readable version should preserve status code, canonical, H1, core body copy, visible offers, key links, schema and update date. It can omit decorative UI. It cannot invent a different page.

4. PrerenderProxy

PrerenderProxy is the productised version of this pattern. Its live homepage positions it as an open, edge-hosted prerender layer for JavaScript apps that serves bot-readable HTML. It states support for verified bot detection, Puppeteer-rendered snapshots, edge-cached snapshots, proactive sitemap warming, per-bot policy and structured observability events.

Use that kind of layer when a full SSR migration is not going to land this quarter. Put the prerender path at the edge, validate bots with more than a user-agent string, warm high-value URLs, and keep the human SPA path intact.

The operational detail is cache hygiene. Bot HTML must preserve real 404 and 500 states, refresh when source content changes, and avoid serving stale rendered errors as successful pages. A prerender layer is useful only when it is observable.

5. SSR versus CSR versus hybrid

SSR and static HTML are the lowest-risk choices for fetch-critical pages. Hybrid SSG/ISR works well for catalogues where freshness is controlled. Prerendering is a bridge for existing SPAs. CSR-only pages are weak when their facts require JavaScript execution.

6. Open Graph and Twitter Cards

Open Graph and card metadata are not magic AI ranking factors. They are summary affordances: stable, compact fields in the first HTML response. Use specific title, meta description, og:title, og:description, canonical and image alt text. Do not use generic site-wide descriptions for pages that need to answer a specific commercial question.

Keep these fields aligned with visible copy. If the H1 says one thing, the OG title says another, and the body waits for JavaScript, the fetcher receives a confused evidence packet. The metadata should summarise the same facts the page actually exposes.

7. The noscript fallback technique

<noscript> is not a replacement for SSR, but it is a cheap safety net for fetch-critical SPAs. Keep it short, equivalent and factual.

<div id="root"></div>
<noscript>
  <main>
    <h1>Fibre broadband for businesses</h1>
    <p>Compare Elisa business fibre plans, address availability and installation options.</p>
    <a href="/business/fibre/availability/">Check address availability</a>
    <a href="/business/fibre/pricing/">View plan pricing</a>
  </main>
</noscript>

8. Token-efficient HTML

Fetch mode has a context budget. Spend it on the evidence packet: H1, direct summary, key facts, labelled tables, canonical link, schema and next actions. Do not make the model wade through mega-menu duplication, skeleton screens, serialised app state, tag managers and cookie modals before the answer.

DOM order matters. Put the main answer before global navigation where possible, keep product grids paginated with real links, and make tables small enough to survive extraction. The model should reach the page-specific facts before it reaches the footer.

const BOT_RE = /(GPTBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User|Googlebot|bingbot)/i;

export async function handleRequest(req) {
  const ua = req.headers.get("user-agent") || "";
  if (!BOT_RE.test(ua)) return fetch(req);

  const renderUrl = new URL("https://prerenderproxy.com/render");
  renderUrl.searchParams.set("url", req.url);
  return fetch(renderUrl, { headers: { "x-original-user-agent": ua } });
}

9. Anti-patterns

The anti-pattern is not JavaScript. The anti-pattern is making JavaScript the only path to facts. Avoid empty app shells, client-only pricing, client-only availability, FAQ answers injected after hydration, OG metadata generated client-side, JSON-LD injected after hydration, infinite scroll without server pagination, and cached bot error pages returned as 200.

Paywalls need special care. If you want AI systems to understand a URL, expose a truthful public summary and the next action. Do not show humans one offer and bots another.

Also avoid measuring only the happy path. Test with the documented user agents, with a plain HTTP client, and with the rendered browser DOM. If those three disagree, fix delivery before changing copy.

10. CTA

If the page matters to AI visibility, make it fetchable first. Start with Lean Render, use PrerenderProxy or the same edge-prerender pattern where a full migration is not available, then measure exactly what each fetcher receives.

The companion post turns this into a scorecard: status codes, raw HTML, rendered DOM, extracted facts, citations and per-vendor readiness. Read Measuring Fetch next.

Paired post: Measuring Fetch

Series: Optimising Memory · Optimising Search · Optimising Fetch · Measuring Memory · Measuring Search · Measuring Fetch

Foundation: AI Knowledge Modes — Memory · Search · Fetch