Talk About Network

Google


Register and Login
Nick
Password
Register create new account Sign up is FREE and you can post replies, new topics, bookmark posts and more!
Recover lost password


Programming > Html > robots.txt and ...
Latest [ Topics | Posts ] Archive Post A New Topic Post a Reply
<< Topic < Post Post 1 of 8 Topic 9869 of 10681
Post > Topic >>

robots.txt and regular expressions?

by Tomasz Chmielewski <tch@[EMAIL PROTECTED] > May 3, 2008 at 01:43 PM

I'm not sure what is the right group for asking questions about 
robots.txt file, so I'm asking it here.

I would like to exclude robots from accessing such links:

/index.php?title=One_page&action=edit
/index.php?title=Other_page&action=edit

What should be a robots.txt line to exclude such pages (for bots which 
understand regexps, like Googlebot, Yahoo Slurp etc.)?

1) Disallow: /index.php*action=edit

2) Disallow: /index\.php.*action=edit


According to http://www.google.com/help/faq_codesearch.html#regexp
(and 
http://en.wikipedia.org/wiki/Regular_expression#Syntax),
it should be 
the 2) one.

However, almost every "robots.txt regexp" search result seem to point to 
the 1) one.

What is the correct answer?



-- 
Tomasz Chmielewski
http://wpkg.org
 




 8 Posts in Topic:
robots.txt and regular expressions?
Tomasz Chmielewski <tc  2008-05-03 13:43:05 
Re: robots.txt and regular expressions?
Sherman Pendley <spamt  2008-05-03 14:38:04 
Re: robots.txt and regular expressions?
faerber.jan@[EMAIL PROTEC  2008-05-03 12:21:51 
Re: robots.txt and regular expressions?
faerber.jan@[EMAIL PROTEC  2008-05-03 12:37:08 
Re: robots.txt and regular expressions?
Nikita the Spider <Nik  2008-05-04 11:12:56 
Re: robots.txt and regular expressions?
faerber.jan@[EMAIL PROTEC  2008-05-04 13:09:26 
Re: robots.txt and regular expressions?
Sherman Pendley <spamt  2008-05-05 05:41:35 
Re: robots.txt and regular expressions?
Nikita the Spider <Nik  2008-05-05 21:01:52 

Post A Reply:
  Go here to Signup

AddThis Feed Button


About - Advertising - Contact - Frequently Asked Questions - Privacy Policy - Terms of Use - Signup

Contact
tan12V112 Thu Oct 16 0:38:56 CDT 2008.