Different robots.txt files?

  • Resolved Alwin

    (@wp-opti)


    2 years ago

    I have installed Yoast SEO on 12 sites, all of them with the latest version of Yoast SEO.

    However, looking at my robots.txt files today I noticed that some of my sites have this text:

    # START YOAST BLOCK
# ---------------------------
User-agent: *
Disallow:

Sitemap: https://www.domain-example/sitemap_index.xml
# ---------------------------
# END YOAST BLOCK

    While other websites still have the “old” setup:

    User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

    So all these websites are running the latest Yoast SEO version, but the robots.txt files are different. How is that happening?

Viewing 5 replies - 1 through 5 (of 5 total)
  • Plugin Support Maybellyne

    (@maybellyne)

    Hello @wp-opti

    Thanks for reaching out regarding your robots.txt file. In v19.0, we updated the plugin to add your XML sitemap link(s) to your robots.txt file?while we changed the copy in the?comment. When a site does not have a user-defined robots.txt file (at the point of installation/activation), Yoast SEO generates a default output:

    START YOAST BLOCK
    ---------------------------
    User-agent: *
    Disallow:
    Sitemap: https://www.example.com/sitemap_index.xml
    ---------------------------
    END YOAST BLOCK

    You mentioned some of your sites running the latest version of the plugin don’t have the robots.txt file in the format above. You can follow this help article to edit the file.

    Thread Starter Alwin

    (@wp-opti)

    Hello Maybellyne,

    Thanks for your fast reply.

    However, my question is how it’s possible that some of my sites are updated to the newest robots.txt and other sites are not, while all are running the latest version of the plugin?

    (I know how to edit the robots.txt file, just like to know)

    Plugin Support Maybellyne

    (@maybellyne)

    Hello Alwin,

    For the sites running the latest version of the plugin but don’t have the robots.txt file in the format above, can you confirm if the file is writable or if your WordPress installation has disabled file editing?

    Thread Starter Alwin

    (@wp-opti)

    I have disabled file editing in my wp-config file like this:
    define( ‘DISALLOW_FILE_EDIT’, true );

    However… this line of code is in all my wp-config files on all my websites!

    So all websites have file editing disabled, but on some websites the Yoast plugin did update the robots.txt file!

    Strange, don’t you think?

    Suwash

    (@suascat_wp)

    You confirmed the given line of code is in wp-config.php file on all your website. We just know having this line of code will not let plugins automatically write/override the files in the server, and so in the case of the Yoast SEO plugin as well. Unfortunately, we are not sure how it became possible in your case. We recommend that you review your server settings.

    Nevertheless, while reviewing the rule in the robots.txt, we would say that it will not hurt your site’s SEO and crawlability as it is an optimized directive — https://yoast.com/ultimate-guide-robots-txt/ that Yoast SEO recommends and writes on the robots.txt file.

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Different robots.txt files?’ is closed to new replies.