Difference between revisions of "Aboutus:Bot"

Latest revision as of 17:33, 18 March 2014

The main job of the AboutUs Bot is to generate basic initial pages and analysis about websites. The bot pulls initial page data once when a page is first created. Website analysis may be pulled multiple times but is cached to prevent continuous access by the bot. We want the bot to be well behaved, if you are seeing otherwise please contact us and let us know.

User-Agent String

The AboutUs Bot User-Agent string contains the following:

AboutUsBot/VERSION (PURPOSE; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)

For example:

AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)

The current AboutUs Bot version is Harpy.

Blocking the AboutUs Bot

Using a robots.txt file, you can choose to not have the About Us Bot access your website. This doesn't mean that we won't create a page for your website. Our members still have the opportunity to contribute their own content describing your site.

To prevent the AboutUs Bot from accessing your site in the future, please include the following lines in your /robots.txt file.

User-agent: AboutUsBot

Disallow: /

The AboutUs Bot will also honor a rule like this in your robots.txt file:

User-agent: *

Disallow: /

However; this rule will prevent all well behaved bots, including Google, from crawling your site.

For more information about robots.txt read our robots.txt article.

@@ Line 1: / Line 1: @@
-{{HelpPageAmbassadorKristina}}
+{{DISPLAYTITLE:The AboutUs Bot}}
-< [[Help FAQ]]
+The main job of the AboutUs Bot is to generate basic initial pages and analysis about websites. The bot pulls initial page data once when a page is first created. Website analysis may be pulled multiple times but is cached to prevent continuous access by the bot. We want the bot to be well behaved, if you are seeing otherwise please [[help/contact|contact us]] and let us know.
-== The AboutUs Bot ==
+==User-Agent String==
+The AboutUs Bot User-Agent string contains the following:
-The main job of the [[AboutUs Bot]] is to create basic pages about websites.  It gathers descriptive public information about a website from several sources (such as the [[whois]] record) to build this [[DomainPage|domain page]].  This pre-built [[WikiWiki|wiki]] page gives website owners and [[AboutUs]] contributors a head-start in creating a useful and informative [[AboutUs.org]] page.  For more information try a [[WikiAnatomy|tour of AboutUs and wiki]].
+:: '''<nowiki>AboutUsBot/VERSION (PURPOSE; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)</nowiki>'''
+For example:
+:: '''<nowiki>AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)</nowiki>'''
-Recently, the AboutUs Bot started marking '''[[parked pages]]'''. If a website was mistakenly marked as parked please [mailto:help@aboutus.org email us] and we will un-do this for you.
+The current AboutUs Bot version is <strong>Harpy</strong>.
-==How it adds a [[WikiPage]] to our site about your site!==
+==Blocking the AboutUs Bot==
-<youtube>pYPFmAErvuw</youtube><br /><br />
+Using a [[Learn/How-To-Use-Robots.txt|robots.txt file]], you can choose to not have the About Us Bot access your website. This doesn't mean that we won't create a page for your website.  Our members still have the opportunity to contribute their own content describing your site.
-== How do I prevent the bot from gathering info about my site ==
+To prevent the AboutUs Bot from accessing your site in the future, please include the following lines in your /robots.txt file.
-Using a '''[[Learn/How-To-Use-Robots.txt|robots.txt''' file]], you can choose not to have your future [[AboutUs.org]] pages initialized with selected content from your website.  '''This doesn't mean that we won't create a Wiki Page for your website.'''  Our members still have the opportunity to contribute their own content describing your site, as well as adding their own [[constructive]] reviews.
-To prevent the [[AboutUs:Bot]] from collecting your site content in the future, please include the following lines in your /robots.txt file.
 :: '''User-agent: AboutUsBot'''
 :: '''Disallow: /'''
-:The [[AboutUs:Bot]] will include the following in it's User-Agent string:
+The AboutUs Bot will also honor a rule like this in your robots.txt file:
-:: '''Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot)'''
-:Please note that the current AboutUs Bot behavior is to visit each site only once to initialize the [[AboutUs.org]] page.
-; Other supported bot prevention methods
-:The [[AboutUs:Bot]] will also honor a rule like this in your robots.txt file:
 :: '''User-agent: *'''
 :: '''Disallow: /'''
-''For more information about [[Learn/How-To-Use-Robots.txt|robots.txt]], read [[Learn/How-To-Use-Robots.txt|this article]].''
+However; this rule will prevent all well behaved bots, including Google, from crawling your site.
-== What about my address? ==
-: Even though your address may be publicly available in various [[Whois services]], if your website has a robots.txt file that denies access to AboutUs:Bot, we will honor your intentions and not publish your contact details on your [[AboutUs.org]] page.
-: Please be aware that if we have already published your address, it was because it was available to us through a popular 3rd party API service.  Your address is probably completely visible in your WHOIS record, and if you want your address to be kept private in the future, you can subscribe to an address protection service through your registrar.
-== How do I remove my [[AboutUs.org]] page? ==
-: Completely erasing [[WikiPage]] content is considered by our editors as a [[TestEdit]] and the page is restored.  If you would like to remove the content created by the bot (Title & Description) and contact details (Address & Contact), please only remove '''the content''' in those sections.  Other sections, including the reviews, thumbnail, language, external links, and contributed content should remain.  See [[No Bot Policy]] for more information.
-[[category:AboutUs Help]]
+For more information about robots.txt read our [[Learn/How-To-Use-Robots.txt|robots.txt]] article.
-[[category:AboutUs FAQ]]
-[[Category:AboutUsBot]]
-__NOTOC__
-__NOEDITSECTION__