Changeset 4868 for webTweet

Timestamp:

Aug 14, 2013, 12:12:39 PM (10 years ago)

Author:

Sam Hocevar

Message:

web: ask crawler bots to hit us less aggressively.

File:

• web/trunk/www/robots.txt (modified) (1 diff)

web/trunk/www/robots.txt

@@ -5 +5 @@
 Disallow: CVS
 Disallow: .svn
+
+# Be gentle
+User-agent: *
+Crawl-delay: 5
 
 # Prevent excessive search engine hits