> ## Knowledge Base Index
> Fetch the complete knowledge base index at: https://support.seotesting.com/sitemap.xml
> Use this file to discover available pages before exploring further.
> Pure-Markdown content can be obtained by appending a '.md' suffix to the content URLs listed in the sitemap (without the trailing slash).

# Robots.txt Monitoring

## **What is robots.txt Monitoring?**

robots.txt Monitoring in SEOTesting automatically watches your site's robots.txt file, stores every version, and alerts you the moment anything changes. 
It's part of SEOTesting's Website Monitor, a low-touch set of tools that watch the files search engines read first.
## **Why use robots.txt Monitoring?**

Accidental robots.txt changes are surprisingly common. A staging config shipped to production, a developer blocking a directory, or a CMS update quietly rewriting the file - any of these can change how search engines crawl your site without anyone realising. Because Search Console reporting lags behind, you might not see the effect for a week or more.

Monitoring your robots.txt means you get the answer to "what changed, and when?" before a ranking drop turns up. This is useful for:

* **Catching accidental blocks** - spot a Disallow: / pushed by mistake before Google recrawls the site.
* **Migrations and replatforms** - confirm robots.txt rules carry over correctly when a site moves CMS or host, with a full audit trail if anything goes wrong.
* **Client and agency work** - know within hours when a client's developer ships a change that rewrites robots.txt, instead of finding out after the organic dip.
* **Diagnosis** - prove to a dev team that a traffic drop started the day a specific rule was added.

## **Where to find robots.txt Monitoring** 

From the Launchpad, select the website at the top of the page that you wish to monitor, then from the left-hand menu select '**Monitor**'.

![](https://storage.crisp.chat/users/helpdesk/website/-/3/4/3/d/343dcfd35d951400/screenshot-2026-08-11-104552_12gmxoi.png =873xauto)
Under '**Website Configuration**' where it says '**Robots.txt**, click on '**View Report**':

![](https://storage.crisp.chat/users/helpdesk/website/-/3/4/3/d/343dcfd35d951400/screenshot-2026-08-11-105439_178dhgd.png =849xauto)

Here you will be able to see every version of your robots.txt:

![](https://storage.crisp.chat/users/helpdesk/website/-/3/4/3/d/343dcfd35d951400/screenshot-2026-08-11-110150_11cj95o.png =899xauto)

You can pick any two dates in the 'Compare' and 'with' options on the top right of the page - this will show you exactly which lines were added, modified, or removed, side by side.

To see further details on any of the changes, click '**Show/hide**' to expand.

### **Email alerts on every change**
You get notified the moment your robots.txt changes, so you can review it, roll it back, or raise a ticket before it affects crawling.

### **Visibility across every site you manage**
Whether you manage ten sites or a hundred, every website in your account is monitored automatically, with all changes surfaced in one place.

|| robots.txt Monitoring is part of SEOTesting's Website Monitor, which also includes the [Sitemap Monitor](https://support.seotesting.com/en/article/sitemap-monitor-13urtmu/).