Moderatobot is a moderator of websites. The main aim of the service is to release webmasters and site moderators from the routine to save a lot of time for more essential affairs.
At its final version Moderatobot will be able to perform constant automatic SEO and content moderation of websites.
There is still much to do, but you can already use Moderatobot for checking Robots.txt files.
SEO moderation of websites
SEO moderation is validation of robotx.txt files, sitemap.xml files, meta tags, links, HTML of websites to indicate various kinds of potential problems which may cause incorrect indexing of your site by search engines.
Check Robots.txt file
By using the Robots.txt file webmaster can control the behavior of search engines robots (crawlers) when indexing his site. The entire site or some its area can be closed from indexing. The robots can be informed about such important information as Sitemap files, expected delay between two subsequent accesses the site, etc.
Errors in robots.txt file and its deviations from the standards can cause problems with the indexing of your websites by some search engines.
Moderatobot checks robots.txt file in order to identify potential problems that search robots might face when processing your robots.txt files. And then offers its own version of the robots.txt file that will best meet the standard and can be equally understood by most major robots.
Robots.txt theory | How to check robots.txt | Start checking
Content moderation of websites
Content moderatiion of websites is checking of syntax errors, detection of the obscene and abusive languages, adults only content.
Moderatobot does not visit any website by its own will. The only reason Moderatobot has visited your site is that you or someone else used it to check the site.
For the time being, Moderatobot only performs robots.txt files validation. But soon it will be able to parse sitemap.xml files and a content of websites.
Access to your website
As robots.txt files are open resources, anyone can access your robots.txt file without permission.
Before using Moderatobot to parse a content of your site, everyone will be obliged to pass through a verification procedure to prove his rights to do so. Besides, you will be able to control the access to your website by Moderatobot using Disallow directives in site’s robots.txt file. For example, you can entirely close your website from visiting of Moderatobot by adding the following group in the robots.txt file:
Note, that you cannot close robots.txt file itself from visiting this way because ‘/robots.txt” is not allowed as a value of Disallow directive.
You can also block any page of your site from being accessed by well behaved robots using meta tag “robots” like this:
<meta name="robots" content="nofollow" />
or block any specific link like this:
<a href="TheLink.aspx" rel="nofollow">The link to be blocked</a>
Moderatobot User-Agent string
The following string represens the Moderatobot:
Mozilla/5.0 (compatible; Moderatobot; +https://www.moderatobot.com/about.aspx)
Note that some fake robots can send the same string to impersonate Moderatobot. Read the next section about how to ensure that the visitor is exactly the Modearatobot.
How to verify Moderatobot
If you want to verify that the true Moderatobot visits your site, it is better to do a reverse DNS lookup, as the IP- addresses from which it can operate may vary, but it still be working under Moderatobot.com domain.
So, to perform the Moderatobot verification, do the following:
1. Check if a User-Agent string of a robot contains “Moderatobot”.
2. Get the IP address of the robot from your server logs.
3. Perform a reverse DNS lookup for the IP-address to determine the domain name of the robot. Verify that the domain name ends with “moderatobot.com”. If it is not – the robot is fake.
4. Perform a forward DNS lookup for the domain name. Verify that the IP is the same as the IP-address of the robot from the server logs. If it is not – the robot is fake.