> ## Documentation Index > Fetch the complete documentation index at: https://docs.aiola.app/llms.txt > Use this file to discover all available pages before exploring further. # AI Crawlers > Identify every crawler Aiola recognizes and how each one is classified in Analytics. This is the reference for which AI systems Aiola can confirm have reached a site. Classification is ordered and first-match-wins against the request User-Agent. ## Dashboard categories | Category | Stored kinds | What it means for the site owner | | -------- | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | Answers | `fetch` | An assistant fetched a page on demand for a live user; this is the category most directly connected to answer-engine visibility. | | Indexing | `search` or `index` | A search or AI-search crawler fetched content for an index that may later power discovery or answers. | | Training | `crawl` | A crawler fetched content for a dataset, model-training, research, or general AI corpus. It does not prove the site appeared in a live answer. | | Other | `ads`, `social`, or any other kind | Advertising and link-preview fetches that do not establish answer, indexing, or training visibility. | `DuckAssistBot` is also mapped to Answers for compatibility with older stored rows. Current ingest classifies it as `fetch`. ## Supported vendors and bots | Vendor | Bots | Dashboard category | What it means for the site owner | | ------------ | ------------------------------------------------------------------------- | ------------------ | ------------------------------------ | | OpenAI | `ChatGPT-User` | Answers | Live assistant fetch observed | | OpenAI | `OAI-SearchBot` | Indexing | Search/index fetch observed | | OpenAI | `GPTBot` | Training | Dataset or training crawl observed | | OpenAI | `OAI-AdsBot` | Other | Advertising or social fetch observed | | Anthropic | `Claude-User` | Answers | Live assistant fetch observed | | Anthropic | `Claude-SearchBot` | Indexing | Search/index fetch observed | | Anthropic | `ClaudeBot`, `anthropic-ai` | Training | Dataset or training crawl observed | | Perplexity | `Perplexity-User` | Answers | Live assistant fetch observed | | Perplexity | `PerplexityBot` | Indexing | Search/index fetch observed | | Mistral | `MistralAI-User` | Answers | Live assistant fetch observed | | Mistral | `MistralAI-Index` | Indexing | Search/index fetch observed | | xAI | `Grok-DeepSearch` | Answers | Live assistant fetch observed | | xAI | `xAI-SearchBot` | Indexing | Search/index fetch observed | | xAI | `GrokBot`, `xAI-Web-Crawler`, `xAI-Grok`, `xAI-Bot`, `Grok` | Training | Dataset or training crawl observed | | Google | `Google-NotebookLM`, `Google-Read-Aloud`, `Google-Agent` or `GoogleAgent` | Answers | Live assistant fetch observed | | Google | `Google-InspectionTool`, `Googlebot` | Indexing | Search/index fetch observed | | Google | `Google-Extended`, `Google-CloudVertexBot`, `GoogleOther` | Training | Dataset or training crawl observed | | Apple | `Applebot` | Indexing | Search/index fetch observed | | Apple | `Applebot-Extended` | Training | Dataset or training crawl observed | | Microsoft | `Copilot` | Answers | Live assistant fetch observed | | Microsoft | `Bingbot` or `bingbot`, `msnbot` | Indexing | Search/index fetch observed | | Amazon | `Amzn-User` | Answers | Live assistant fetch observed | | Amazon | `Amzn-SearchBot` | Indexing | Search/index fetch observed | | Amazon | `Amazonbot` | Training | Dataset or training crawl observed | | ByteDance | `TikTokSpider` | Indexing | Search/index fetch observed | | ByteDance | `Bytespider`, `Doubaobot` | Training | Dataset or training crawl observed | | Meta | `Meta-ExternalFetcher` or `meta-externalfetcher` | Answers | Live assistant fetch observed | | Meta | `meta-webindexer` | Indexing | Search/index fetch observed | | Meta | `Meta-ExternalAgent`, `FacebookBot`, or `meta-externalagent` | Training | Dataset or training crawl observed | | Meta | `meta-externalads`, `facebookexternalhit` | Other | Advertising or social fetch observed | | Common Crawl | `CCBot` | Training | Dataset or training crawl observed | | Cohere | `cohere-ai`, `cohere-training-data-crawler` | Training | Dataset or training crawl observed | | DuckDuckGo | `DuckAssistBot` | Answers | Live assistant fetch observed | | DuckDuckGo | `DuckDuckBot` | Indexing | Search/index fetch observed | | You.com | `YouBot` | Indexing | Search/index fetch observed | | Moonshot AI | `Kimi-User` | Answers | Live assistant fetch observed | | Moonshot AI | `Kimi-SearchBot` | Indexing | Search/index fetch observed | | Moonshot AI | `KimiBot` | Training | Dataset or training crawl observed | | Alibaba | `Qwen-User` | Answers | Live assistant fetch observed | | Alibaba | `QwenBot`, `TongyiBot`, `AliyunBot` | Training | Dataset or training crawl observed | | Baidu | `Baiduspider` | Indexing | Search/index fetch observed | | Baidu | `ERNIEBot`, `YiyanBot` | Training | Dataset or training crawl observed | | Zhipu AI | `ChatGLM-Spider` | Training | Dataset or training crawl observed | | DeepSeek | `DeepSeekBot` | Training | Dataset or training crawl observed | | Allen AI | `AI2Bot` | Training | Dataset or training crawl observed | | Diffbot | `Diffbot` | Training | Dataset or training crawl observed | | Timpi | `Timpibot` | Training | Dataset or training crawl observed | | ImageSift | `ImagesiftBot` | Training | Dataset or training crawl observed | | Liner | `LinerBot` | Training | Dataset or training crawl observed | | Qualified | `QualifiedBot` | Training | Dataset or training crawl observed | | Yandex | `YandexBot` | Indexing | Search/index fetch observed | | Seznam | `SeznamBot` | Indexing | Search/index fetch observed | | Petal | `PetalBot` | Indexing | Search/index fetch observed | | Qwant | `Qwantbot` | Indexing | Search/index fetch observed | Aiola re-derives the bot, vendor, and kind from `ua`; callers cannot choose their classification. A User-Agent that does not match this table returns HTTP `204` and is not stored. See the [API Reference](/reference/api#ingest-bot-and-crawler-analytics) for the ingest payload and limits. ## AI Visibility Score The server computes a deterministic `0`–`100` score from five components. Each component is rounded before the points are summed and the result is clamped to `0`–`100`. | Component | Maximum | Formula | | ------------------- | ------: | --------------------------------------------------------------------------------------------------------- | | Crawl coverage | 30 | `30 × crawled popular pages / popular pages`, capped at 30 | | AI crawler breadth | 20 | `20 × distinct AI crawlers / 8`, capped at 20 | | AI answer referrals | 25 | `25 × visitors from AI answers / 25`, capped at 25 | | Assistant breadth | 15 | `15 × distinct referring assistants / 5`, capped at 15 | | Discovery files | 10 | `10 × served discovery types / 3`; the types are `robots.txt`, `llms.txt` or `llms-full.txt`, and sitemap | The score is returned as `aiVisibility` by the authenticated bot-traffic/analytics overview data and is used by the AI Visibility automation and MCP analytics overview. The current Analytics AI renderer shows crawler summaries, categories, pages, broken pages, discovery files, and filters, but does not render the numeric score as a dashboard card. It is not a claim about model training or whether a vendor will cite the site; it measures observed crawls, observed AI referrals, and discovery-file coverage. ## What detection proves | Observation | What you can conclude | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | A row in Answers | An identified assistant fetcher requested the recorded URL. | | A row in Indexing | An identified search/index crawler requested the recorded URL. | | A row in Training | An identified corpus or training crawler requested the recorded URL. | | No row for a vendor | Aiola did not receive a matching server-side beacon in the selected period; this does not prove the vendor cannot access the site. | | A `200` status | The capture point reported success or omitted status, which defaults to `200`. | | A `404`–`599` status | The capture point reported that error response to the crawler. |