---
title: SEO Crawlers
description: Separate search engine and SEO auditing bot activity from AI assistant and crawler visits.
---

Use **SEO Crawlers** to see automated requests from search engines and SEO tools. PromptEye keeps these separate from AI Traffic because a search engine crawl and an AI assistant fetching a page have different purposes. A rise in one should not be read as a rise in the other.

## Google Search Console

If Search Console is connected to the project, this tab includes Google search clicks, impressions, click-through rate (CTR), average position, daily trends, top queries, and top pages. These numbers describe Google Search performance, not crawler requests. For example, a bot may crawl a page without anyone clicking it in Google results.

The query and page lists help connect search visibility to content: impressions with few clicks may be a reason to review the title or snippet, while a page with clicks can be worth comparing with its AI crawl activity. Search Console can omit some queries and data may not be available immediately for a new connection or date range.

If Search Console is not connected, the page offers a connection flow. Choose the Google property that matches the project website; choosing a different property would make the report describe another site. See [Settings and setup](../setup/) for connecting it.

## Crawler activity and crawl health

**Top SEO Crawlers** ranks the identified search engine and SEO audit bots by their requests in the selected period. Use it to check whether search engines are reaching the site or whether an audit tool is scanning it. These visits do not represent human visitors.

**SEO crawl health** groups successful requests, redirects, 4xx errors, and 5xx errors. Repeated 4xx responses can point to missing or blocked pages; frequent redirects can make crawlers take extra steps; repeated server errors or slow responses can make crawling less reliable. Check the path, status, and last-seen time before deciding something needs fixing: a redirect to a replacement URL can be intentional.

Before reading a high error rate as a problem with your site, check how much of it was automated probing. Scanners request files that exist on no site, such as environment files or cloud credentials, and take a well-known crawler name while doing it. A line under the summary counts every probing request in the period, and a **Vulnerability scanning** filter narrows the issue table to the ones that failed or redirected, so the count is usually the larger of the two. The totals and thresholds do not change; separating them tells you whether search engines are struggling to reach your pages or whether a stranger is simply trying doors that were never there.

## Sitemap coverage

Connect a public sitemap for a page-by-page comparison between URLs it lists and the bot visits PromptEye has recorded. The report groups URLs as recently visited, stale, never visited, or a gap/new URL. Adjust the stale threshold to decide how many days without a visit counts as stale, and use status, bot, or URL filters to find a specific group.

**Sync now** asks PromptEye to read the sitemap again, so newly added or removed URLs can be reflected. **Export CSV** downloads the complete sitemap report, including URLs not loaded into the current table page and regardless of the current table filters; apply filters in your spreadsheet if you need a narrower handoff. **Remove** removes this sitemap configuration and its coverage report for the project. You can connect it again later.

The sitemap URL must be publicly accessible and use the same domain as the project. A private, blocked, invalid, or different-domain URL cannot be connected. A sitemap shows what URLs are listed; it does not force a bot to visit them or prove they were indexed.