Robots.txt resources
Robots exclusion standard (Wikipedia)
Robots.txt Specifications (Google)
Using robots.txt (Yandex)
A Standard for Robot Exclusion (Robotstxt.org)
A Method for Web Robots Control (Robotstxt.org)
An Extended Standard for Robot Exclusion (Sean Connor)
Useful tools
Create and test robots.txt (Google)
Analyze robots.txt (Yandex)
Robots.txt theory | 

How to check robots.txt

 | Start checking 

Domain name

Enter а domain name for which you are performing the verification of а robots.txt file. Do not append the trailing '/robots.txt' or '/' to the domain name. By default Moderatobot uses 'http' protocol and 80 port to access your domain. So you need to specify the protocol or/and port only if you use another ones.

Some examples of correct domain names:

  • example.com
  • www.example.com
  • http://example.com
  • http://www.example.com
  • https://www.example.com
  • http://www.example.com:8080

Load and check

Load and check

Click the "Load and check button. Moderatobot will search for the robots.txt file in the root directory of your domain and, if found, place the content of the robots.txt into the left 'Source' pane and then perform it's checking. After checking and adjusting to the standards, the recommended content of the robots.txt will be placed into the right 'Target' pane.

Check

Check

You can also edit the text you want to verify in the left 'Source' pane manually. Then click the "Check" button.

Settings

Settings

Clicking the "Settings" button will open the "Settings" dialog where you can select a checking mode. After you change the mode and click "OK" in the dialog, Moderatobot will perform checking of a text in the left 'Source' pane  according to new settings.

Checking mode

Every time you check the content of the robots.txt Moderatobot use one of these modes:

  • Thorough - if you want to know what problems might face bots which strictly follow the standards when processing your robots.txt file, use the "Thorough" mode. In this mode Moderatobot warns of any deviations from the standards and the situations that different bots may interpret differently. And recommends a content that will best meet the standards and can be equally understood by most major bots. The default mode.

  • Optimistic - Moderatobot assumes that the most bots are able to pass correctly through such situations as unknown or non standard directives or "*" and "$" characters, etc. in a robots.txt file. And notifies only about rough errors in a file's location, accessibility and format.

Save the result

Save

You can save the result file that Moderatobot recommends after checking by clicking the "Save" button which located just above the right 'Target' pane. Or simply select and copy to the clipboard the result text.

Checking errors, warnings and notes

After checking Moderatobot displays messages about errors and other possible problems that bots can face when processing your robots.txt file at the bottom of the page. "Thorough" mode will display detailed information whereas 'Optimistic' mode - only error messages. Just change the mode by clicking the "Settings" button if you want to check the file in different conditions.

We appreciate every opinion which can help to improve the service.
Name
Email
Subject
Text
 
 
Start checking