Virtual robots.txt

Try uploading a robots.txt to the root level of your domain (this is where this file goes).

https://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360

I know how to create one. What I wasn’t sure on was if I could modify the output of the virtual robots.txt that WP uses.

Check your dashboard privacy settings and try an actual robots.txt, which have priority over any “virtual” one.

@iridiax thanks! It was the privacy settings preventing google from crawling my website… Cheers