Title: SoftGlaze AI Crawler Monitor
Author: softglaze
Published: <strong>18. August 2026</strong>
Last modified: 25. Oktober 2026

---

Plugins suchen

![](https://ps.w.org/softglaze-ai-crawler-monitor/assets/banner-772x250.png?rev=
3652913)

![](https://ps.w.org/softglaze-ai-crawler-monitor/assets/icon-256x256.png?rev=3652913)

# SoftGlaze AI Crawler Monitor

 Von [softglaze](https://profiles.wordpress.org/softglaze/)

[Herunterladen](https://downloads.wordpress.org/plugin/softglaze-ai-crawler-monitor.0.7.1.zip)

 * [Details](https://de.wordpress.org/plugins/softglaze-ai-crawler-monitor/#description)
 * [Rezensionen](https://de.wordpress.org/plugins/softglaze-ai-crawler-monitor/#reviews)
 *  [Installation](https://de.wordpress.org/plugins/softglaze-ai-crawler-monitor/#installation)
 * [Entwicklung](https://de.wordpress.org/plugins/softglaze-ai-crawler-monitor/#developers)

 [Support](https://wordpress.org/support/plugin/softglaze-ai-crawler-monitor/)

## Beschreibung

SoftGlaze AI Crawler Monitor records every request from a known AI crawler that 
reaches WordPress. It shows you which crawlers came, what they asked for, and what
your site told them at the time.

Most plugins in this area stop at writing a rule. This one is built around the gap
between writing a rule and the rule working, because on the open web those are not
the same thing.

You can find out:

 * Which AI crawlers requested pages on your site, how often, and which pages.
 * What each crawler was told, and whether it took any notice.
 * Which requests were refused, if you chose to refuse them.

#### What it can and cannot do

There are three ways to tell a crawler to stay away. Only one of them is enforced,
and the plugin repeats that on every screen where you make the choice.

 1. A robots.txt rule. This is a request, not a barrier. Well behaved crawlers honour
    it. One that ignores it can still fetch your pages, and nothing on your server 
    stops it.
 2. An `X-Robots-Tag: noai, noimageai` header, sent only on requests from that crawler.
    Also a request, and a weaker one, because `noai` is not an agreed standard and 
    most crawlers ignore it.
 3. Refusing the request with a 403. This is the only setting that is actually enforced.
    A matching request gets an empty „403 Forbidden“ and the page is never sent.

The header never contains `noindex`, `nofollow` or `none` under any setting. Those
would take your pages out of ordinary Google and Bing results, which is not what
anyone means when they say they want to block AI.

Three crawlers in the list are documented by their own operators as fetching pages
when a person asks a question, with robots.txt rules that may not apply to them:
ChatGPT-User, Perplexity-User and Meta-ExternalFetcher. All three are marked on 
the crawler screen, right next to the setting, so you find out before you rely on
it rather than afterwards.

Your robots.txt is always served, even to a crawler you are refusing. Blocking a
crawler from reading robots.txt would leave it no way to learn that you want it 
to stay away.

#### Per-content control

The site-wide setting is the default. Any single post, page or product can override
it from the edit screen, from quick edit, or from bulk edit. A column in the posts
list shows which content has its own setting, and you can filter by it.

A per-post setting sends the `X-Robots-Tag` header and does nothing else. It adds
no lines to robots.txt, it cannot refuse a request, and it cannot lift a refusal
you set site-wide. The edit screen says so in those words.

#### What it cannot see

The plugin records requests that reach WordPress. Plenty of requests never get that
far, and rather than quietly under-report, it tells you so:

 * Requests your CDN or firewall answers before they reach your server.
 * Requests served from a page cache. On a test server with page caching on, only
   one request in five reached WordPress at all.
 * Static files. Images, stylesheets, scripts and fonts are served without running
   WordPress.
 * Addresses on your domain that WordPress does not handle.
 * Any crawler that is not in this plugin’s list.

There is a banner above every figure in the plugin saying this. You cannot dismiss
it, because it changes what the numbers mean.

#### Crawler identity checking

A user agent is self declared, so anything can claim to be GPTBot. The plugin can
check whether a crawler really is what it says.

Both checks are switched off until you turn them on, because both mean contacting
a server outside your site. Left off, the plugin still does everything else, and
crawlers are recorded as „not checked“, which is what they honestly are.

A crawler is only ever recorded as having failed a check when a check actually ran
and gave a negative answer. A check that could not run is never treated as a failure.

Checking only works where the operator publishes something to check against. Eleven
of the crawlers in the list can be checked. The rest stay „not checked“ however 
you set these options, because their operators publish nothing to check them with,
and the crawler screen tells you which is which.

#### Crawlers included in this version

GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-
Extended, PerplexityBot, Perplexity-User, meta-externalagent, Meta-WebIndexer, Meta-
ExternalFetcher, Applebot, Applebot-Extended, Bytespider, CCBot, Amazonbot, Amzn-
SearchBot, Amzn-User, cohere-ai, Diffbot, Diffbot-User, Timpibot, YouBot.

Every one of these was checked against the operator’s own published documentation,
and each entry records the page it came from. The crawler screen links to that page
for each one.

Three of them have no published documentation at all: Bytespider, cohere-ai and 
Timpibot. They are in the list because the requests are real and you are entitled
to see them, and each is labelled on screen as undocumented, since nobody has stated
whether they read robots.txt.

Crawler names and robots.txt tokens change over time. The list ships with the plugin,
it is versioned, and it is never fetched from a remote server.

#### Other plugins by SoftGlaze

 * [SoftGlaze PDF Invoices for WooCommerce](https://wordpress.org/plugins/softglaze-pdf-invoices/)—
   invoices, packing slips, proformas and credit notes from your WooCommerce orders,
   rendered locally.
 * [SoftGlaze Maintenance Mode & Coming Soon](https://wordpress.org/plugins/softglaze-maintenance-mode-coming-soon/)—
   coming soon and maintenance pages with templates, countdowns and access control.
 * [SoftGlaze Click to Chat](https://wordpress.org/plugins/softglaze-click-to-chat/)—
   a WhatsApp chat button with multiple agents, schedules and WooCommerce support.

### External services

**On a default install this plugin contacts nothing at all.**

It has two features that make an outbound request. Both are switched off when the
plugin is installed, and neither one does anything until you tick its box on the
settings screen. Neither runs while a page is being served; both run only on a scheduled
task in the background.

There is no analytics, no telemetry, no licence check and no update check. Nothing
is ever sent to SoftGlaze or to the plugin’s author, under any setting.

**1. Reverse address lookups (DNS) — off by default**

What it is and what it is for: a reverse DNS lookup, used to confirm that a crawler’s
address really belongs to the company its user agent claims. A user agent is self
declared, so anything can claim to be a given crawler; this is how that claim is
tested.

Where it goes: this does not contact a service the plugin chooses. It asks the domain
name resolver your server is already configured to use, which will be your host’s
or your network’s. Because the plugin does not select the provider, there is no 
third party service here with terms or a privacy policy for us to link to. Which
resolver your server uses, and that operator’s terms, are a matter between you and
your host.

What data is sent, and when: the crawler’s IP address, and nothing else. This is
the only feature in the plugin that sends any visitor information anywhere. It is
sent on a scheduled task that runs at most every fifteen minutes, and only for addresses
belonging to crawler requests already recorded on your site. The address of an ordinary
human visitor is never looked up, because an ordinary visitor is never recorded 
in the first place.

Under which conditions: only when you have switched „Check crawler identity by reverse
address lookup“ on. With it off, crawlers are recorded as „not checked“ and no address
goes anywhere.

Which crawlers this affects: one, YouBot, because you.com is the only operator in
the current list that publishes a hostname pattern to check an address against. 
For every other crawler in the list this setting does nothing at all.

**2. Published address lists (HTTPS) — off by default**

What it is and what it is for: some AI companies publish, on their own website, 
the list of IP addresses their crawlers use. When you switch this on, the plugin
downloads those files and checks recorded crawler addresses against them. This is
the second, independent way of testing a self declared user agent.

What data is sent, and when: nothing about your site, nothing about you, and nothing
about any visitor. It is an ordinary unauthenticated GET request for a public file—
the same request your own browser would make if you typed the address into it. The
request’s User-Agent names the plugin and its version, and nothing else: no site
URL, no identifier and no query string is attached. Each file is downloaded at most
once per operator per day and the result is stored on your server.

Under which conditions: only when you have switched „Check crawler identity against
published address lists“ on.

These, in full, are the only addresses this setting ever requests. There are no 
others, and the plugin never contacts an address that is not in this list.

_OpenAI_, for GPTBot, OAI-SearchBot and ChatGPT-User.
 Files requested: https://
openai.com/gptbot.json , https://openai.com/searchbot.json , https://openai.com/
chatgpt-user.json Terms of use: https://openai.com/policies/terms-of-use/ Privacy
policy: https://openai.com/policies/privacy-policy/

_Anthropic_, for ClaudeBot, Claude-User and Claude-SearchBot.
 File requested: https://
claude.com/crawling/bots.json Terms of use: https://www.anthropic.com/legal/consumer-
terms Privacy policy: https://www.anthropic.com/legal/privacy

_Apple_, for Applebot.
 File requested: https://search.developer.apple.com/applebot.
json Terms of use: https://www.apple.com/legal/internet-services/ Privacy policy:
https://www.apple.com/legal/privacy/

_Perplexity_, for PerplexityBot and Perplexity-User.
 Files requested: https://www.
perplexity.ai/perplexitybot.json , https://www.perplexity.ai/perplexity-user.json
Terms of use: https://www.perplexity.ai/hub/legal/terms-of-service Privacy policy:
https://www.perplexity.ai/hub/legal/privacy-notice

_Common Crawl_, for CCBot.
 File requested: https://index.commoncrawl.org/ccbot.
json Terms of use: https://commoncrawl.org/terms-of-use Privacy policy: https://
commoncrawl.org/privacy-policy

**Documentation links, which are not requests**

Separately from the two features above, the Crawler access screen shows a „documentation“
link beside each crawler, pointing at the page on the operator’s own website where
that crawler’s behaviour is documented. These are ordinary links in the page. The
plugin never fetches them, on any setting, and nothing is requested unless you choose
to click one and open it in your own browser.

They are listed here so that the presence of these addresses in the plugin’s source
is not mistaken for an undisclosed service. The pages linked are Amazon’s Amazonbot
page, Anthropic’s crawler support article, Apple’s Applebot page, Common Crawl’s
CCBot page, Diffbot’s robots.txt FAQ, Google’s crawlers page, Meta’s web crawlers
page, OpenAI’s bots page, Perplexity’s bots page and you.com’s YouBot page. Three
crawlers — Bytespider, cohere-ai and Timpibot — have no documentation link, because
their operators publish none.

Every address in this section is also recorded in the plugin folder at `includes/
data/bot-registry.php`, where each crawler carries the documentation page it was
checked against and the address list published for it, if there is one. The registry
ships with the plugin and is never fetched from a remote server.

### Privacy

Visitor IP addresses are never stored in the request log. Each record holds a one
way hash instead, made with a random value that is generated when the plugin is 
installed on your site and is never shared. The hash cannot be turned back into 
an address.

The raw user agent is not stored either. It is matched against the crawler list,
recorded as a crawler name, and then discarded.

There is one exception, and it only exists if you switch on reverse address lookups:

 * A crawler’s full address is held in a separate table for as long as the lookup
   takes, then deleted straight away.
 * Anything left over is removed after 60 minutes whether it was checked or not,
   including on sites where scheduled tasks are not running.
 * Deactivating the plugin empties that table immediately.
 * It never appears on any screen or in any export.

With reverse address lookups off, which is the default, that table is never written
to at all.

How long things are kept is not the same for everything, and the settings screen
spells this out:

 * Individual request records: 90 days by default, adjustable between 7 and 365.
 * Daily totals per crawler: kept indefinitely. Counts only, with no visitor information.
 * Daily list of most requested addresses: 400 days. Counts and addresses only.
 * Identity check results: 7 days.
 * The short lived lookup queue described above: 60 minutes at most.

There is no analytics and no telemetry. Nothing about your site or your visitors
is ever sent to the plugin’s author.

## Screenshots

[⌊The dashboard. Every known AI crawler that reached WordPress, how many requests
each made, its share of the total, and how many days it was seen.⌉⌊The dashboard.
Every known AI crawler that reached WordPress, how many requests each made, its 
share of the total, and how many days it was seen.⌉[

The dashboard. Every known AI crawler that reached WordPress, how many requests 
each made, its share of the total, and how many days it was seen.

[⌊Crawlers you told to stay away. The left panel counts requests that arrived after
you had already disallowed that crawler; the right panel counts requests your server
refused with a 403. Both count only from the day you set the rule.⌉⌊Crawlers you
told to stay away. The left panel counts requests that arrived after you had already
disallowed that crawler; the right panel counts requests your server refused with
a 403. Both count only from the day you set the rule.⌉[

Crawlers you told to stay away. The left panel counts requests that arrived after
you had already disallowed that crawler; the right panel counts requests your server
refused with a 403. Both count only from the day you set the rule.

[⌊Most requested URLs, filterable by crawler. The notice explains why this ranking
is approximate and points at the URL checker for exact figures.⌉⌊Most requested 
URLs, filterable by crawler. The notice explains why this ranking is approximate
and points at the URL checker for exact figures.⌉[

Most requested URLs, filterable by crawler. The notice explains why this ranking
is approximate and points at the URL checker for exact figures.

[⌊Crawler access. One setting per crawler, each with the operator's own documentation
linked and the date it was checked. ChatGPT-User carries a warning because OpenAI
documents that robots.txt may not apply to it.⌉⌊Crawler access. One setting per 
crawler, each with the operator's own documentation linked and the date it was checked.
ChatGPT-User carries a warning because OpenAI documents that robots.txt may not 
apply to it.⌉[

Crawler access. One setting per crawler, each with the operator’s own documentation
linked and the date it was checked. ChatGPT-User carries a warning because OpenAI
documents that robots.txt may not apply to it.

[⌊What each setting actually does, and what it cannot do. Only "Refuse the request"
is enforced; the other two are requests a crawler is free to ignore.⌉⌊What each 
setting actually does, and what it cannot do. Only "Refuse the request" is enforced;
the other two are requests a crawler is free to ignore.⌉[

What each setting actually does, and what it cannot do. Only „Refuse the request“
is enforced; the other two are requests a crawler is free to ignore.

[⌊The URL checker. Exact per-address figures counted from individual records, the
crawlers that requested the address after being asked not to, and what that address
currently tells each crawler.⌉⌊The URL checker. Exact per-address figures counted
from individual records, the crawlers that requested the address after being asked
not to, and what that address currently tells each crawler.⌉[

The URL checker. Exact per-address figures counted from individual records, the 
crawlers that requested the address after being asked not to, and what that address
currently tells each crawler.

[⌊Settings. Retention is 90 days by default. Both identity checks contact a server
outside your site, so both are off until you switch them on.⌉⌊Settings. Retention
is 90 days by default. Both identity checks contact a server outside your site, 
so both are off until you switch them on.⌉[

Settings. Retention is 90 days by default. Both identity checks contact a server
outside your site, so both are off until you switch them on.

[⌊The posts list gains an "AI access" column and a filter, so you can see which 
content has its own setting.⌉⌊The posts list gains an "AI access" column and a filter,
so you can see which content has its own setting.⌉[

The posts list gains an „AI access“ column and a filter, so you can see which content
has its own setting.

[⌊Per-content control from quick edit. A per-post setting sends the advisory header
only — it adds nothing to robots.txt and cannot refuse a request.⌉⌊Per-content control
from quick edit. A per-post setting sends the advisory header only — it adds nothing
to robots.txt and cannot refuse a request.⌉[

Per-content control from quick edit. A per-post setting sends the advisory header
only — it adds nothing to robots.txt and cannot refuse a request.

## Installation

 1. Install through Plugins, Add New, Upload Plugin, or copy the folder to `/wp-content/
    plugins/`.
 2. Activate it on the Plugins screen.
 3. Open „AI Crawlers“ in the admin menu.

Nothing is blocked and nothing is disallowed when you activate it. The plugin starts
by recording what happens and changes nothing about your site until you pick a setting,
so you can look before you decide anything.

On activation it writes a small helper file into `wp-content/mu-plugins/` so it 
can identify a crawler at the earliest point WordPress allows. It removes that file
again when you deactivate the plugin. If the folder cannot be written to, the plugin
detects slightly later in the page load instead and tells you which mode is running.
Nothing is lost either way.

## FAQ

### Does this stop AI companies using my content?

No, and no plugin can honestly promise that.

Two of the three mechanisms are requests a crawler is free to ignore. The third 
refuses matching requests at your server and is enforced, but it matches on the 
user agent the request claims, and a user agent can be faked. A scraper that does
not announce itself is unaffected. This is not a firewall.

What you get instead is evidence. If a crawler kept coming after you told it not
to, you will see it.

### Why do my numbers look low?

Almost certainly page caching. When a page comes from a cache, WordPress never runs,
so the request is not recorded. The plugin has no way to know how much it is missing,
so it says that plainly instead of guessing.

### Why does the top pages list say it is approximate?

Because it is. Daily totals keep the 100 most requested addresses per crawler per
day, so a page that never reaches a daily top 100 stays invisible in a longer list
no matter how many requests it collects. Every row links to the URL checker, which
counts individual records and is exact.

### Can I submit my site to ChatGPT or Claude?

No. Neither company offers a submission form, an endpoint or an API, and any plugin
that says otherwise is describing something that does not exist.

What OpenAI documents is that any public website can appear in ChatGPT search, and
that the way to be eligible is simply not to block their search crawler. Claude’s
web search runs on a third party search index rather than one Anthropic takes submissions
to.

So the only lever you have is the one this plugin already gives you, which is whether
a crawler is allowed. That is why the crawler screen warns you before you disallow
a crawler that feeds a product which links back to its sources.

### What happens to my data if I delete the plugin?

Nothing is deleted unless you tick the box on the settings screen first. Deactivating
the plugin never deletes anything.

### Does it work on multisite?

Per site. Activate it on each site individually. Network activation is refused with
a message explaining why, because everything the plugin stores belongs to one site.

## Rezensionen

Zu diesem Plugin liegen noch keine Rezensionen vor.

## Mitwirkende und Entwickler

„SoftGlaze AI Crawler Monitor“ ist Open-Source-Software. Folgende Menschen haben
an diesem Plugin mitgewirkt:

Mitwirkende

 *   [ Azhar Ali ](https://profiles.wordpress.org/softglaze/)

[Übersetze „SoftGlaze AI Crawler Monitor“ in deine Sprache.](https://translate.wordpress.org/projects/wp-plugins/softglaze-ai-crawler-monitor)

### Interessiert an der Entwicklung?

[Durchstöbere den Code](https://plugins.trac.wordpress.org/browser/softglaze-ai-crawler-monitor/),
sieh dir das [SVN-Repository](https://plugins.svn.wordpress.org/softglaze-ai-crawler-monitor/)
an oder abonniere das [Entwicklungsprotokoll](https://plugins.trac.wordpress.org/log/softglaze-ai-crawler-monitor/)
per [RSS](https://plugins.trac.wordpress.org/log/softglaze-ai-crawler-monitor/?limit=100&mode=stop_on_copy&format=rss).

## Änderungsprotokoll

#### 0.7.1

 * The brand name is now spelled SoftGlaze, as in the other SoftGlaze plugins. The
   robots.txt block markers keep their earlier spelling so blocks already written
   are still found and replaced.
 * The published address list download no longer puts your site’s address in its
   User-Agent. It now sends the plugin name and version only, as the External services
   section says.
 * Removed the Plugin URI from the plugin header until the plugin’s page on softglaze.
   com is live, and set the Author to SoftGlaze.
 * Tested up to WordPress 7.1.

#### 0.7.0

 * Renamed to Softglaze AI Crawler Monitor. The previous name described the plugin
   rather than identifying it, so it now leads with the Softglaze name. Every identifier
   moved with it, including the database tables, so this version installs as a new
   plugin rather than upgrading the old one.
 * The helper file in `wp-content/mu-plugins/` no longer assumes the plugin folder’s
   name when locating the plugin. It uses the real path recorded when the file was
   written, and the plugin rewrites that path by itself if the folder is ever moved
   or renamed.
 * The „External services“ section of this readme now states, for each outbound 
   request, exactly what is sent, when, and under what condition, and separates 
   the operator documentation links shown on screen — which the plugin never fetches—
   from the addresses it actually requests.

#### 0.6.0

 * Per-content control. Set AI access on a single post, page or product from the
   edit screen, quick edit or bulk edit, with a filterable column in the posts list.
 * The URL checker now reports conflicts, meaning which crawlers requested an address
   after being asked not to, and which were refused.
 * The URL checker also shows what each crawler is currently told about that one
   address, and whether the address is reachable at all.
 * Six more crawlers, each checked against its operator’s documentation: Meta-WebIndexer,
   Meta-ExternalFetcher, Applebot, Amzn-SearchBot, Amzn-User and Diffbot-User.
 * The crawler screen now warns you before you disallow a crawler that feeds a product
   which links back to its sources, because there is no way to submit your site 
   to those products afterwards.

#### 0.5.0

 * Every crawler in the list checked against its operator’s own documentation, with
   the source page recorded and linked from the crawler screen.
 * Crawlers whose operators say robots.txt may not apply to them are now marked 
   as such, next to the setting.
 * Crawlers whose operators publish no documentation at all are now marked as such.
 * Identity checking now works wherever an operator publishes a way to check, covering
   ten crawlers by published address list or reverse DNS.
 * FacebookBot removed, because Meta’s crawler documentation no longer lists it.
 * Applebot-Extended corrected to a robots.txt name only, which is what Apple documents
   it as. It could never have appeared in reports, and it no longer looks as though
   it might.

#### 0.4.0

 * Violation reporting, with separate panels for crawlers that ignored your robots.
   txt and requests that were refused.
 * Crawler identity checking by reverse address lookup and published address lists.
   Both switched off by default.
 * Optional removal of all data when the plugin is deleted. Off by default.

#### 0.3.0

 * Choose what each crawler is told: allow, monitor, or disallow.
 * robots.txt rules and the `X-Robots-Tag` header for disallowed crawlers.
 * Optional refusal of matching requests with a 403.
 * Detection and handling of an existing robots.txt file at the site root.

#### 0.2.0

 * Dashboard showing requests per crawler over 7, 30 and 90 days, with a trend and
   the most requested addresses.
 * URL checker for exact per-address counts.
 * Cache state banner.

#### 0.1.0

 * Crawler detection and request recording.
 * Daily totals and retention cleanup.

## Meta

 *  Version **0.7.1**
 *  Zuletzt aktualisiert **vor 3 Wochen**
 *  Aktive Installationen **weniger als 10**
 *  WordPress-Version ** 6.2 oder höher **
 *  Getestet bis **7.1.4**
 *  PHP-Version ** 7.4 oder höher **
 *  Sprache
 * [English (US)](https://wordpress.org/plugins/softglaze-ai-crawler-monitor/)
 * Schlagwörter
 * [ai bots](https://de.wordpress.org/plugins/tags/ai-bots/)[AI Crawlers](https://de.wordpress.org/plugins/tags/ai-crawlers/)
   [block ai](https://de.wordpress.org/plugins/tags/block-ai/)[robots.txt](https://de.wordpress.org/plugins/tags/robots-txt/)
 *  [Erweiterte Ansicht](https://de.wordpress.org/plugins/softglaze-ai-crawler-monitor/advanced/)

## Bewertungen

Es wurden noch keine Rezensionen eingereicht.

[Deine Rezension](https://wordpress.org/support/plugin/softglaze-ai-crawler-monitor/reviews/#new-post)

[Alle Rezensionen anzeigen](https://wordpress.org/support/plugin/softglaze-ai-crawler-monitor/reviews/)

## Mitwirkende

 *   [ Azhar Ali ](https://profiles.wordpress.org/softglaze/)

## Support

Möchtest du etwas mitteilen? Brauchst du Unterstützung?

 [Support-Forum anzeigen](https://wordpress.org/support/plugin/softglaze-ai-crawler-monitor/)

## Spenden

Möchtest du die Weiterentwicklung dieses Plugins unterstützen?

 [ Für dieses Plugin spenden ](https://softglaze.com/)