robots.txt

@dyrer

You can use the virtual robots.txt file generated by the plugin or you could create your own physical one, if you need more control of the content in it.

@lukecavanagh Thank you, but I get 404 when click on robots.txt from SEO Framework settings

@dyrer

What options do you have enabled in SEO > Robots Meta Settings?

yes

Thanks for dropping by again Luke :).

Athanasiadis, to answer your questions:

1.
Nope, you should not have a custom robots file: Robots are handled by WordPress and Google encourages to leave as much out of it as possible.
Currently, you’re advised to use the “robots” meta tag. This is a very advanced tag and the most important aspects are completely covered through various Robots settings throughout The SEO Framework.

From the Google Webmasters Blog, I quote:

If you use both a robots.txt file and robots meta tags
If the robots.txt and meta tag instructions for a page conflict, Googlebot follows the most restrictive. More specifically:

• If you block a page with robots.txt, Googlebot will never crawl the page and will never read any meta tags on the page.
• If you allow a page with robots.txt but block it from being indexed using a meta tag, Googlebot will access the page, read the meta tag, and subsequently not index it.

The latter allows you to ask Googlebot to follow the Canonical URL or redirect as well — this lets more of your site be discovered and allows you to gain more link juice.

For this very reason alone, I discourage a manual robots.txt file.

2.
Unfortunately, a plugin or theme is messing up your permalinks, badly.

To fix this (temporarily), please re-save your permalinks settings. Settings -> Permalinks -> Save Changes. The virtual robots.txt file outputted by WordPress and adjusted by The SEO Framework should work again.

To fix this (permanently), please discover any outdated plugins or themes (with 2+ years of no updates) and address them by deactivation.
From there, you should check if any active plugins require the Rewrite Engine and which aren’t popular (less than 100k installations). Plugins of this type include (to name a few) sitemap, custom post type, registration, forum, search, and optimization plugins.

Be sure to keep your software up to date as well.

I can’t help you any further with this as it’s extremely complex, time consuming and might even require back-end (code) access. I also had this issue before (which caused a redirect loop upon login) and I had to dive into a (for me) must-have (multisite) plugin and rewrite a whole class within — this took over a year to discover.

If you’re still encountering this issue I must ask you to post it on the www.remarpro.com general help forums. Note that the output of the virtual robots.txt file is done by WordPress itself, so the volunteers over there should be able to help you :).

3.
The best practice is to use the default robots SEO settings.
Throughout these forums this question has been asked quite often; and covering this sole topic is good for at least a 12 page manuscript.
There’s also lot to be found on Moz and other SEO bloggers about which tag to use where.
If you could give me a sample or a web page URL I might be able to lead you into the right direction; however, the default settings are most likely always the best option.

I hope this all helps you out in some way or another. Cheers! ??

• This reply was modified 8 years, 1 month ago by Sybre Waaijer.
• This reply was modified 8 years, 1 month ago by Sybre Waaijer.

Thanks for you detailed answer and you cover all my questions
But my main problem is how to make nginx server create robots.txt, so I cant have 404 error
I have tested you plugin with apache and works perfect
If you have more tips to advice, please share

or something like that

rewrite ^/robots\.txt$ /static/robots.txt last;

Hi Athanasiadis,

The robots.txt file is outputted by WordPress core, The SEO Framework merely filters its contents.
So I find it weird that there are any issues, at all.

Nevertheless, I believe the code is:

rewrite ^/robots\.txt$ /index.php?robots=1 last;

With that code you should get the dynamic robots.txt file.

If you wish to have a static file, you simple have to upload a real robots.txt file in your website’s root folder (something like: example.com/www, example.com/public or serveruname/public_html). The rewrite rule shouldn’t be present, I believe.

I hope this helps! Cheers ??

Thank you the last line works perfect