• Lookign through my server logs, I see a lot of requests for index.rdf and rss.xml. I’m sure these are coming from some search bot.
    Can anyone give me details on what I might do to my robots.txt file or a meta entry to direct these requests to the appropriate WP files?
    TIA

Viewing 6 replies - 1 through 6 (of 6 total)
  • wont copying wp-rss.php as rss.xml work?

    I have search bots looking for files which just do NOT exist on my site ….. not so much a case of just indexing the contents, it seems they have their own agenda too.
    Bad enough they steal the content you do have, it would seem they are also ‘ripping open the mattresses’.

    Naming feeds with filenames like index.rdf and rss.xml is a common habit, so the bot is probably just guessing.
    If you like, you can use an Apache .htaccess redirect to tell them where they can really be found, like so:
    Redirect permanent /index.xml /wp-rss.php
    Redirect permanent /rss.xml /wp-rss.php
    A good and polite bot will get the hint from the ‘permanent’ directive. A crappy bot will not get the hint and will keep guessing, but it should still find the page anyway.

    Thread Starter randybrown

    (@randybrown)

    raena – I tried your suggestion. Placed those two lines at the bottom of my .htaccess file. But it caused 500 errors.
    Could there be another method, or a typo?
    TIA

    hm, what about mod rewrite? not the best solution but definately a quick fix if redirects arn’t working.

    the reason for google searching for this files is rumored to be becoz google is apparently planning for a blog search engine… ??

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘How to direct bots to correct files?’ is closed to new replies.