More about Moderatobot
Privacy Policy
Terms of use
Latest news
Hello World
August 24, 2016

Welcome to Moderatobot - automatic websites moderator...
Subscribe to news

About Moderatobot

 

Introduction

Moderatobot is a moderator of websites. The main aim of the service is to release webmasters and site moderators from the routine to save a lot of time for more essential affairs.

At its final version Moderatobot will be able to perform constant automatic SEO and content moderation of websites.

There is still much to do, but you can already use Moderatobot for checking Robots.txt files.

SEO moderation of websites

SEO moderation is validation of robotx.txt files, sitemap.xml files, meta tags, links, HTML of websites to indicate various kinds of potential problems which may cause incorrect indexing of your site by search engines.

Check Robots.txt file

By using the Robots.txt file webmaster can control the behavior of search engines robots (crawlers) when indexing his site. The entire site or some its area can be closed from indexing. The robots can be informed about such important information as Sitemap files, expected delay between two subsequent accesses the site, etc.

Errors in robots.txt file and its deviations from the standards can cause problems with the indexing of your websites by some search engines.

Moderatobot checks robots.txt file in order to identify potential problems that search robots might face when processing your robots.txt files. And then offers its own version of the robots.txt file that will best meet the standard and can be equally understood by most major robots.

Robots.txt theory  |  How to check robots.txt  |  Start checking

Content moderation of websites

Content moderatiion of websites is checking of syntax errors, detection of the obscene and abusive languages, adults only content.

Moderatobot crawler

Moderatobot does not visit any website by its own will. The only reason Moderatobot has visited your site is that you or someone else used it to check the site.

For the time being, Moderatobot only performs robots.txt files validation. But soon it will be able to parse sitemap.xml files and a content of websites.

Access to your website

As robots.txt files are open resources, anyone can access your robots.txt file without permission.

Before using Moderatobot to parse a content of your site, everyone will be obliged to pass through a verification procedure to prove his rights to do so. Besides, you will be able to control the access to your website by Moderatobot using Disallow directives in site’s robots.txt file. For example, you can entirely close your website from visiting of Moderatobot by adding the following group in the robots.txt file:

User-agent: Moderatobot
Disallow: /

Note, that you cannot close robots.txt file itself from visiting this way because ‘/robots.txt” is not allowed as a value of Disallow directive.

You can also block any page of your site from being accessed by well behaved robots using meta tag “robots” like this:

<meta name="robots" content="nofollow" />

or block any specific link like this:

<a href="TheLink.aspx" rel="nofollow">The link to be blocked</a>

Moderatobot User-Agent string

The following string represens the Moderatobot:

Mozilla/5.0 (compatible; Moderatobot; +https://www.moderatobot.com/about.aspx)

Note that some fake robots can send the same string to impersonate Moderatobot. Read the next section about how to ensure that the visitor is exactly the Modearatobot.

How to verify Moderatobot

If you want to verify that the true Moderatobot visits your site, it is better to do a reverse DNS lookup, as the IP- addresses from which it can operate may vary, but it still be working under Moderatobot.com domain.

So, to perform the Moderatobot verification, do the following:

1. Check if a User-Agent string of a robot contains “Moderatobot”.

2. Get the IP address of the robot from your server logs.

3. Perform a reverse DNS lookup for the IP-address to determine the domain name of the robot. Verify that the domain name ends with “moderatobot.com”. If it is not – the robot is fake.

4. Perform a forward DNS lookup for the domain name. Verify that the IP is the same as the IP-address of the robot from the server logs. If it is not – the robot is fake.

We appreciate every opinion which can help to improve the service.
Name
Email
Subject
Text