can i run my site with no robots.txt file?

It’s not crucial, but it can be beneficial. Can you paste your robots.txt here?

Hi there. Sorry I dropped the ball on this post. Actually I finally figured out what was happening, that there was a Disallow for my root folder. Here is my robots.txt.

Anyway since then there has been a different problem, which is that no my webnaster tools are showing a crawl error with all of my post url’s with /trackback/ at the end of the url. for example: http//mydomain.com/trackbacks/

It lists this as a crawl error for every single post on my site, saying these url’s are restricted by robots.txt.

I don’t have trackbacks active on my site so not sure what is up with this. Here is my robots.txt in case you still are willing to take a look:

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Allow: /wp-content/uploads

You’re seeing the error because you have Disallow: /trackback in your robots.txt file. Even if Trackbacks are disabled, the link may be included somewhere in your theme. Typically, Google does not list trackbacks URLs in search results, but the existence of both the link and Disallow: /trackback will trigger an error nonetheless.

In this case, I’d either not worry about the error or remove the Disallow: /trackback line.

Also, you should consider removing the Allow: /wp-content/uploads line. It’s actually redundant. Since /wp-content/uploads is not disallowed by the above lines, it’s automatically assumed by search engine bots to be an allowed directory.

Hey thanks for your help with this. I made the changes you suggest. Much appreciated ??

You’re welcome!