The AboutUs Bot
The main job of the AboutUs Bot is to generate basic pages and analysis about websites.
How do I prevent the bot from gathering info about my site
Using a robots.txt file, you can choose not to have your future AboutUs pages initialized with selected content from your website or analyzed. This doesn't mean that we won't create a Wiki Page for your website. Our members still have the opportunity to contribute their own content describing your site, as well as adding their own reviews.
To prevent the AboutUs Bot from accessing your site in the future, please include the following lines in your /robots.txt file.
- User-agent: AboutUsBot
- Disallow: /
- The AboutUs Bot will include the following in it's User-Agent string:
- AboutUsBot/VERSION (PURPOSE; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)
For example:
- AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)
The current AboutUs Bot version is Harpy.
- Please note that the current AboutUs Bot behavior is to visit each site only once to initialize the page.
- Other supported bot prevention methods
- The AboutUs Bot will also honor a rule like this in your robots.txt file:
- User-agent: *
- Disallow: /
For more information about robots.txt, read this article.
What about my address?
- Even though your address may be publicly available in various Whois services, if your website has a robots.txt file that denies access to AboutUs:Bot, we will honor your intentions and not publish your contact details on your AboutUs page.
- Please be aware that if we have already published your address, it was because it was available to us through a popular 3rd party API service. Your address is probably completely visible in your WHOIS record, and if you want your address to be kept private in the future, you can subscribe to an address protection service through your registrar.
How do I remove my AboutUs page?
- Completely erasing page content is considered by our editors to be a TestEdit and the page will be restored. If you would like to remove the content created by the bot (Title & Description) and contact details (Address & Contact), please only remove the content in those sections. Other sections, including the reviews, thumbnail, language, external links, and contributed content should remain. See No Bot Policy for more information.