AI Crawlers

This is the reference for which AI systems Aiola can confirm have reached a site. Classification is ordered and first-match-wins against the request User-Agent.

Dashboard categories

Category	Stored kinds	What it means for the site owner
Answers	`fetch`	An assistant fetched a page on demand for a live user; this is the category most directly connected to answer-engine visibility.
Indexing	`search` or `index`	A search or AI-search crawler fetched content for an index that may later power discovery or answers.
Training	`crawl`	A crawler fetched content for a dataset, model-training, research, or general AI corpus. It does not prove the site appeared in a live answer.
Other	`ads`, `social`, or any other kind	Advertising and link-preview fetches that do not establish answer, indexing, or training visibility.

DuckAssistBot is also mapped to Answers for compatibility with older stored rows. Current ingest classifies it as fetch.

Supported vendors and bots

Vendor	Bots	Dashboard category	What it means for the site owner
OpenAI	`ChatGPT-User`	Answers	Live assistant fetch observed
OpenAI	`OAI-SearchBot`	Indexing	Search/index fetch observed
OpenAI	`GPTBot`	Training	Dataset or training crawl observed
OpenAI	`OAI-AdsBot`	Other	Advertising or social fetch observed
Anthropic	`Claude-User`	Answers	Live assistant fetch observed
Anthropic	`Claude-SearchBot`	Indexing	Search/index fetch observed
Anthropic	`ClaudeBot`, `anthropic-ai`	Training	Dataset or training crawl observed
Perplexity	`Perplexity-User`	Answers	Live assistant fetch observed
Perplexity	`PerplexityBot`	Indexing	Search/index fetch observed
Mistral	`MistralAI-User`	Answers	Live assistant fetch observed
Mistral	`MistralAI-Index`	Indexing	Search/index fetch observed
xAI	`Grok-DeepSearch`	Answers	Live assistant fetch observed
xAI	`xAI-SearchBot`	Indexing	Search/index fetch observed
xAI	`GrokBot`, `xAI-Web-Crawler`, `xAI-Grok`, `xAI-Bot`, `Grok`	Training	Dataset or training crawl observed
Google	`Google-NotebookLM`, `Google-Read-Aloud`, `Google-Agent` or `GoogleAgent`	Answers	Live assistant fetch observed
Google	`Google-InspectionTool`, `Googlebot`	Indexing	Search/index fetch observed
Google	`Google-Extended`, `Google-CloudVertexBot`, `GoogleOther`	Training	Dataset or training crawl observed
Apple	`Applebot`	Indexing	Search/index fetch observed
Apple	`Applebot-Extended`	Training	Dataset or training crawl observed
Microsoft	`Copilot`	Answers	Live assistant fetch observed
Microsoft	`Bingbot` or `bingbot`, `msnbot`	Indexing	Search/index fetch observed
Amazon	`Amzn-User`	Answers	Live assistant fetch observed
Amazon	`Amzn-SearchBot`	Indexing	Search/index fetch observed
Amazon	`Amazonbot`	Training	Dataset or training crawl observed
ByteDance	`TikTokSpider`	Indexing	Search/index fetch observed
ByteDance	`Bytespider`, `Doubaobot`	Training	Dataset or training crawl observed
Meta	`Meta-ExternalFetcher` or `meta-externalfetcher`	Answers	Live assistant fetch observed
Meta	`meta-webindexer`	Indexing	Search/index fetch observed
Meta	`Meta-ExternalAgent`, `FacebookBot`, or `meta-externalagent`	Training	Dataset or training crawl observed
Meta	`meta-externalads`, `facebookexternalhit`	Other	Advertising or social fetch observed
Common Crawl	`CCBot`	Training	Dataset or training crawl observed
Cohere	`cohere-ai`, `cohere-training-data-crawler`	Training	Dataset or training crawl observed
DuckDuckGo	`DuckAssistBot`	Answers	Live assistant fetch observed
DuckDuckGo	`DuckDuckBot`	Indexing	Search/index fetch observed
You.com	`YouBot`	Indexing	Search/index fetch observed
Moonshot AI	`Kimi-User`	Answers	Live assistant fetch observed
Moonshot AI	`Kimi-SearchBot`	Indexing	Search/index fetch observed
Moonshot AI	`KimiBot`	Training	Dataset or training crawl observed
Alibaba	`Qwen-User`	Answers	Live assistant fetch observed
Alibaba	`QwenBot`, `TongyiBot`, `AliyunBot`	Training	Dataset or training crawl observed
Baidu	`Baiduspider`	Indexing	Search/index fetch observed
Baidu	`ERNIEBot`, `YiyanBot`	Training	Dataset or training crawl observed
Zhipu AI	`ChatGLM-Spider`	Training	Dataset or training crawl observed
DeepSeek	`DeepSeekBot`	Training	Dataset or training crawl observed
Allen AI	`AI2Bot`	Training	Dataset or training crawl observed
Diffbot	`Diffbot`	Training	Dataset or training crawl observed
Timpi	`Timpibot`	Training	Dataset or training crawl observed
ImageSift	`ImagesiftBot`	Training	Dataset or training crawl observed
Liner	`LinerBot`	Training	Dataset or training crawl observed
Qualified	`QualifiedBot`	Training	Dataset or training crawl observed
Yandex	`YandexBot`	Indexing	Search/index fetch observed
Seznam	`SeznamBot`	Indexing	Search/index fetch observed
Petal	`PetalBot`	Indexing	Search/index fetch observed
Qwant	`Qwantbot`	Indexing	Search/index fetch observed

Aiola re-derives the bot, vendor, and kind from ua; callers cannot choose their classification. A User-Agent that does not match this table returns HTTP 204 and is not stored. See the API Reference for the ingest payload and limits.

AI Visibility Score

The server computes a deterministic 0–100 score from five components. Each component is rounded before the points are summed and the result is clamped to 0–100.

Component	Maximum	Formula
Crawl coverage	30	`30 × crawled popular pages / popular pages`, capped at 30
AI crawler breadth	20	`20 × distinct AI crawlers / 8`, capped at 20
AI answer referrals	25	`25 × visitors from AI answers / 25`, capped at 25
Assistant breadth	15	`15 × distinct referring assistants / 5`, capped at 15
Discovery files	10	`10 × served discovery types / 3`; the types are `robots.txt`, `llms.txt` or `llms-full.txt`, and sitemap

The score is returned as aiVisibility by the authenticated bot-traffic/analytics overview data and is used by the AI Visibility automation and MCP analytics overview. The current Analytics AI renderer shows crawler summaries, categories, pages, broken pages, discovery files, and filters, but does not render the numeric score as a dashboard card. It is not a claim about model training or whether a vendor will cite the site; it measures observed crawls, observed AI referrals, and discovery-file coverage.

What detection proves

Observation	What you can conclude
A row in Answers	An identified assistant fetcher requested the recorded URL.
A row in Indexing	An identified search/index crawler requested the recorded URL.
A row in Training	An identified corpus or training crawler requested the recorded URL.
No row for a vendor	Aiola did not receive a matching server-side beacon in the selected period; this does not prove the vendor cannot access the site.
A `200` status	The capture point reported success or omitted status, which defaults to `200`.
A `404`–`599` status	The capture point reported that error response to the crawler.

Getting Started

Workspaces & Central

Chat & AI

Project Operations

Monitoring

Connections

Settings

Setup Guides

Reference

Dashboard categories

Supported vendors and bots

AI Visibility Score

What detection proves

​Dashboard categories

​Supported vendors and bots

​AI Visibility Score

​What detection proves

Dashboard categories

Supported vendors and bots

AI Visibility Score

What detection proves