Yoast generated robots.txt not in file directory

  • Resolved Conor

    (@jrocbaby)


    9 months ago

    Hi

    I’m having some problems with the robots.txt file.

    I want to disallow my website from being indexed as it is under development. I’m using yoast SEO which appears to have generated the robots.txt file. however, i cant edit it from within the plugin.

    If i create a robots.txt file in the file manager its not overwriting the one that yoast SEO is generating?

    I have seen on the wordpress support there is a line of code that disallows file editing.

    define( 'DISALLOW_FILE_EDIT', true );

    However this line of code doesn’t exist where it should be to prevent file editing? can anyone help me with this, i seem to be stuck in a loop!

    The page I need help with: [log in to see the link]

Viewing 4 replies - 1 through 4 (of 4 total)
  • azibyaqoob

    (@azibyaqoob)

    Hi,

    If you’re unable to edit the robots.txt file through Yoast SEO due to it being unwritable or if file editing is disabled in your WordPress installation, you’ll need to make adjustments at the server level. In such instances, WordPress generates a virtual robots.txt file if a physical file isn’t present in the site’s root directory. To override this virtual file, follow these steps to create a physical robots.txt file:

    1. Use your favorite text editor and create a text file.
    2. Save the empty file with the name robots.txt.
    3. Upload the file to your server. If you’re unsure where on your server to upload the file, please contact your webhost.

    I hope this helps.

    Thread Starter Conor

    (@jrocbaby)

    Hi,

    I have uploaded a new robots.txt file but that is my issue, it’s not overwriting the generated one

    Plugin Support Maybellyne

    (@maybellyne)

    Hello @jrocbaby,

    Thanks for reaching out about your robots.txt file.

    I want to disallow my website from being indexed as it is under development.

    You could use the feature in WordPress > Settings > Reading > Search Engine Visibility to Discourage search engines from indexing this site while it is still in development

    I’m using yoast SEO which appears to have generated the robots.txt file. however, i cant edit it from within the plugin.

    Yoast SEO includes the ability to create and edit the robots.txt file for your site when your site meets the following requirements:

    • Enable file editing, which is the default option for most WordPress installs
    • Set server-level permissions to allow writing to files, specifically the robots.txt file.

    As @azibyaqoob rightly stated, if you do not see the option to create or edit the file, WordPress has disabled file editing, or your server doesn’t allow writing to the file. Your web host can help you enable these features if you want to use Yoast SEO to edit the robots.txt file.

    I have uploaded a new robots.txt file but that is my issue, it’s not overwriting the generated one

    Can you share the generated one and the uploaded one so I can see the difference?

    Plugin Support Maybellyne

    (@maybellyne)

    This thread was marked resolved due to a lack of activity, but you’re always welcome to re-open the topic. Please read this post before opening a new request.

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Yoast generated robots.txt not in file directory’ is closed to new replies.