The finding that surprised us: access is not the problem. Almost nobody is blocking the crawlers that recommend them. The gap is that most product pages contain almost no facts worth repeating.
There is a growing market for tools that monitor whether AI assistants recommend your store, priced from about $79 to $399 a month. Before building anything else in that direction, we wanted to know how bad the underlying problem actually is. So we measured it.
On 28 August 2026 we fetched, for each of 70 consumer brands: robots.txt, the public
product listing, and one real product page. We parsed robots.txt with correct group precedence
(a crawler obeys its own group and ignores User-agent: * when it has one), then
analysed the raw served HTML — not the rendered DOM — because most AI crawlers
do not execute JavaScript.
The store list was fixed before any results were seen, so the sample could not be selected toward a conclusion. The scanner is public, so is the raw data, and it shares its logic with the free checker so the two cannot disagree.
Sample and limits, stated plainly. This is a convenience sample of well-known, well-resourced brands, not a random sample of Shopify. It is biased toward competence: small stores are likely to do worse, not better. 3 hosts refused our research user-agent outright (403/429) and are excluded from the figures rather than counted as failures. Robots.txt is also not the whole story — a firewall can block AI crawlers in ways no robots.txt reveals. Each figure below prints its own denominator, because they genuinely differ.
Zero of 62 readable robots.txt files blocked any AI search crawler (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Claude-User). Not one. If you have been told your store might be invisible to ChatGPT because of a robots.txt mistake, that is worth checking once — it takes about thirty seconds — but it is not where the problem is.
2 of 62 stores block at least one training crawler
— GPTBot, ClaudeBot, Google-Extended, CCBot
and similar. That is a different decision with a different consequence: it opts you out of model
training without affecting whether an assistant can find and recommend you today.
beardbrand.com — opts out of Amazonbotwww.manscaped.com — opts out of GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagentThis distinction is the single most misunderstood thing in the category. Blocking
GPTBot does not remove you from ChatGPT's recommendations;
OAI-SearchBot is the token that does. Only 11 of 62
stores had written any explicit AI-crawler rule at all.
88% of the 51 product pages we could analyse carried a
Product or ProductGroup node in the served HTML. Of those,
95% had a complete offer — price,
priceCurrency and availability all present. That is the machine-readable
core of "what is this, what does it cost, can I buy it", and most stores have it.
Weaker spots: brand present on 71%,
aggregateRating on 27%. Ratings are the one an
assistant is most likely to want when it compares you to somebody else.
1 of 51 pages contained the string
application/ld+json only inside JavaScript: the structured data is constructed after
load, so it is perfect in dev tools and absent from what a non-JS crawler receives.
3 pages served JSON-LD that failed to parse, which consumers skip entirely.
1 of 49 pages had a <title> that contradicted their own
og:title: a bath towel product page whose title announced a sheet set. The visible
heading was correct, so nothing looks wrong to a human.
That figure is deliberately conservative. Earlier versions of this check flagged
four pages, then three, then two — and each time the extra flags turned out to be legitimate
variation rather than defects: a store using a different internal product name from its display
title, a title carrying a bracketed suffix containing a |, and an SEO-style title
sitting beside a variant-specific og:title ("Women's Dasher NZ Sneakers …" vs
"Women's Dasher NZ – Blizzard/Deep Navy"). The check now fires only when the two share no
leading words at all. Telling a healthy store it is broken is a more expensive error than
missing a marginal case.
These failures share a property: every tool an owner would normally check with runs JavaScript, so all of them report success.
Median readable text in raw HTML was 1508 words, which is healthy. But the median number
of concrete measurements — a number with a unit, like 6.7 oz or
23.5 mm — was 2, and 78% of pages
had fewer than five.
This is the finding we would act on. An assistant asked to compare two products repeats whatever it can attribute. "Holds 120 lb" survives that trip; "premium quality" does not, because it is true of the whole category and distinguishes nothing. A page can be perfectly crawlable, perfectly structured, and still give an assistant nothing to say about you.
| Thinnest raw HTML in the sample | Words | Page size | Product schema |
|---|---|---|---|
| www.starface.world | 149 | 272 KB | no |
| www.leesa.com | 215 | 15 KB | no |
| www.brooklinen.com | 264 | 1639 KB | no |
| www.peakdesign.com | 319 | 240 KB | yes |
| www.mejuri.com | 422 | 1039 KB | yes |
Page size against word count is the tell: a megabyte of HTML containing a few hundred readable words means the content is inside script payloads, waiting for JavaScript that an AI crawler will not run.
49 of 70 stores in this sample expose a live agent-commerce API. Not a page for a crawler to read — an endpoint an AI agent can call.
Shopify is serving /llms.txt files headed "Agent Instructions" that advertise the
Universal Commerce Protocol. Behind them, verified live rather than
read from documentation:
| Endpoint | What it returns |
|---|---|
GET /.well-known/ucp | Merchant profile, UCP version 2026-04-08, service endpoints |
POST /api/ucp/mcp | JSON-RPC tools/list → 13 tools |
The tools include search_catalog, create_cart,
create_checkout, complete_checkout and get_order. An agent
can find a product, build a cart and complete a purchase without parsing a single line of your
HTML.
This reframes everything above. Sections 1 to 4 measure whether a crawler can read you. This measures whether an agent can buy from you, and they are independent: one store in this sample has genuinely broken markup — structured data injected by JavaScript, a title naming a different product — and a perfectly functioning commerce endpoint. Its HTML is invisible; its checkout is not.
A live endpoint does not mean any agent can buy. Probing
further: tools/list answers anonymously, but every tool call is refused
without a published UCP agent profile URI (invalid_profile_url), verified on three
independent stores. Capability enumeration is open; transacting is not. So these stores are
reachable by identified agents, not anonymous ones — an accountability layer, and a
qualification our first phrasing did not make. We did not create an agent profile to test past
it.
41 stores serve Shopify's "Agent Instructions" format; 10 serve
the original descriptive llms.txt convention, which is a plain-language summary and
makes no commerce claim. Zero advertised commerce that did not answer — an
earlier version of our scanner reported four, which turned out to be our own detection bug
rather than anyone's broken endpoint. Note also that no AI vendor currently documents reading
other sites' llms.txt, so treat publishing one as cheap and harmless rather than as
a proven visibility lever. Our probe was read-only: it called tools/list, which
enumerates capabilities, and never created a cart or started a checkout.
In order, cheapest first:
Run these checks on your store, free →
Or one command, against your live store, nothing uploaded:
curl -sO https://raw.githubusercontent.com/krisdiallo/ecom-agent/main/aivis.py
python3 aivis.py yourstore.comSingle file, no dependencies. There is also a
GitHub Action if you
want it to fail your build, and
crawlers.json
— the sourced registry behind every crawler claim on this page, with each entry carrying the
vendor's own wording and the date it was checked. MIT; use it in your own tool.
This study opened by noting that Perplexity, asked where to buy merino base layers, answered with REI, Smartwool, Icebreaker, Amazon and Alpinetrek — five incumbents, no independents. We let that imply AI shopping answers favour big brands.
Checked against a second engine, that does not hold. Google's AI Mode, asked to recommend merino wool sneaker brands on 28 August 2026, returned Allbirds, Giesswein, Woolloomooloo and Merinos — mostly small specialists, one of them an independent Shopify store.
So the honest statement is: it varies enormously by engine, and one engine is not a measurement. That is what our own method notes say, and we had not applied it to our own headline.
We then ran our checker against the independent store Google recommended. It allows all seven search crawlers, declares a sitemap, exposes a live agent-commerce endpoint with 13 tools, and carries 4,126 words of readable text — but its structured data fails to parse. It was recommended anyway, over brands with clean markup.
The tell is in Google's own citations for that answer: a Reddit thread and an independent comparison blog. Which is exactly the thing this page lists below as the factor we cannot measure. On this evidence it is not a footnote — third-party mentions look more decisive than anything on your own pages. The mechanical checks above are the floor, and the floor is apparently lower than we implied.
It does not show that fixing any of this causes recommendations. We measured what crawlers can read, not what assistants choose to say, and we know of no public evidence establishing that link. It cannot see the factor most likely to dominate: whether independent third-party sources describe you consistently. And it is a snapshot of 70 large brands on one day, not a trend.
The honest counterweight: conventional search still handles the overwhelming majority of shopping queries. This is a channel worth being present in, not one to rebuild a store around.
Mostly no. In a scan of 70 well-known consumer brands on 28 August 2026, 0 of 62 readable robots.txt files blocked an AI search crawler such as OAI-SearchBot or PerplexityBot. 2 blocked at least one AI training crawler such as GPTBot or ClaudeBot, which is a different decision and does not affect whether an assistant can recommend the store.
No. GPTBot is OpenAI's training crawler. OAI-SearchBot is the one that surfaces sites in ChatGPT's search results. Blocking GPTBot opts you out of model training while leaving your recommendations intact. The two are separate robots.txt tokens with opposite consequences.
Specificity, far more often than access. Across 51 product pages the median number of concrete measurements — a number with a unit, like 6.7 oz or 23.5 mm — was 2. 78% of pages had fewer than five. Assistants repeat facts; a page built from adjectives gives them nothing to repeat.
Yes, and it is the most deceptive failure of the set. If your JSON-LD is injected by JavaScript after the page loads, it will look correct in browser dev tools and in testers that execute JavaScript, while being entirely absent from the HTML a non-JS crawler receives. 1 of 51 pages scanned had this shape.