How to disable the automatically generated robots.txt?

  • Hi,
    I would like no robots.txt at all. How can i properly disable the automatically generated robots.txt from wordpress ?

    I already deleted the do_robots function in /wp-includes/functions.php, nevertheless, i have a php error when i go to revspa.com/robots.txt

    I d like to get a clean 404 error.

    thank you

Viewing 1 replies (of 1 total)
  • Thread Starter bloup

    (@bloup)

    I have found a solution myself, without modifying functions.php
    You just need to add
    redirect 404 /robots.txt to .htaccess and then remove the /robots.txt using URL removal tool from google webmaster tools.

    I am doing so because i think robots.txt is counter-productive. I noticed i lost traffic each time i added disallowed folders to robots.txt
    I don’t have documents to hide.

    On the other hand, matt cutt said we should not block folders where there is .js
    Then, .swf are not indexed by wordpress.
    So i think it is useless to block /wp-includes/ or /wp-content/

    At last, i think it is useless to block /wp-admin/ and /wp-includes/ because spam bots don’t read robots.txt anyways.

Viewing 1 replies (of 1 total)
  • The topic ‘How to disable the automatically generated robots.txt?’ is closed to new replies.