Four original datasets on whether AI assistants can read, cite and buy from ecommerce storefronts. All MIT licensed, all machine-readable, all regenerable from the scanners in the repository. No signup, no email, no rate limit.
Each figure below is read from the dataset files themselves when this page is built, so the page and the data cannot drift apart.
Every known AI crawler user-agent (165 tokens), classified by what blocking it in robots.txt actually costs a website. Blocking a training crawler removes you from no product recommendations; blocking a search crawler removes you from AI answers in ChatGPT, Perplexity, Claude, Alexa and Siri. Most published robots.txt advice conflates the two. 17 tokens are classified as removing you from AI answers and 38 as training-only opt-outs. Critically, 105 of 165 are marked 'undetermined' because no public source establishes the consequence — they are deliberately not defaulted to a guess. Each row carries a 'basis' field recording evidence strength, so consumers can filter to vendor-documented rows only. Derived in part from ai-robots-txt/ai.robots.txt (MIT).
21 AI crawler user-agent tokens, each tied to the operating vendor's own published documentation. Every entry records the vendor, the product the crawler serves, what blocking it does, whether it honours robots.txt, a verbatim quote from the vendor, a source URL, and the date the source was checked. Entries that could not be tied to a first-party statement carry an explicit unverified flag rather than being presented as sourced. Built to correct a common error: that blocking GPTBot removes a site from ChatGPT's recommendations, when OAI-SearchBot is the token that governs that.
70 consumer ecommerce storefronts probed for a live agent-commerce endpoint an AI agent could call to search a catalogue, build a cart and complete a checkout without parsing HTML. 49 of 70 exposed one. The notable result: all 49 exposed an identical 13-tool surface on a single protocol version, with zero variation across independent brands — indicating agent-commerce capability is currently a property of the ecommerce platform rather than a per-merchant configuration choice. Probing further established that tool enumeration answers anonymously while every tool call is refused without a published agent profile URI, so these stores are transactable by identified agents, not anonymous ones. The probe was read-only throughout and never created a cart or began a checkout.
Raw measurements from 70 consumer brand storefronts, assessing whether AI assistants can actually read them. 62 robots.txt files were parsed with correct group precedence and 51 product pages analysed from raw HTML rather than the rendered DOM, because most AI crawlers do not execute JavaScript. Headline findings: 0 of 62 stores blocked an AI search crawler and 2 blocked a training crawler, so accidental invisibility — the fear the AI-SEO tooling category is sold on — did not occur. 45 of 51 pages carried Product or ProductGroup structured data. The real deficit is specificity: the median product page contains 2 concrete measurements, and assistants repeat attributable facts rather than adjectives. Includes per-host structured-data completeness, offer fields, word counts, title/og:title agreement and JavaScript-injection detection.
# every crawler whose blocking removes you from AI answers, vendor-documented only
curl -s https://raw.githubusercontent.com/krisdiallo/ecom-agent/main/crawler-consequences.json \
| python3 -c "import json,sys; d=json.load(sys.stdin); \
print(*[c['token'] for c in d['crawlers'] \
if c['basis']=='vendor-documented' \
and c['blocking_effect']=='removes_from_ai_answers'], sep='\n')"The crawler registries record evidence strength per row. 105
of 165 crawlers
are marked undetermined rather than guessed, because no public source establishes
what blocking them costs. Filter on basis to choose your own confidence threshold.
The surveys are a snapshot of 70 well-known consumer brands on one date, not a random sample of all ecommerce. They describe what those storefronts served to a non-JavaScript client on 2026-08-28. They cannot tell you whether an assistant will recommend a given store — nobody can, since those rankings are not public.