Sitemap generated without default WordPress elements

  • Resolved Justin Freid

    (@justinfreid)


    9 months ago

    Even if the following is added in the Reading settings,

    User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

    it is left out of the resulting robots.txt file.

    If nothing is added, and only Google News Sitemap generation is on, reference to the sitemap is created but the robots.txt is still missing the above so it is broken. If other elements are added after the above in the Reading settings such as 

    User-agent: GPTBot
Disallow: /

    that code is at the top of robots.txt and the default is still missing.

    I have repeated this on more than 5 sites. Yoast’s block gets properly added to the virtual robots.txt file on all of the sites.

Viewing 1 replies (of 1 total)
  • Plugin Author Rolf Allard van Hagen

    (@ravanh)

    Hi, this plugin does not alter the default WP robots.txt output. It only appends to it. So adding 

    User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

    to the robots.txt field on Settings > Reading should result in having that default content twice in your robots.txt

    I’ve verified and it appears that Yoast SEO is actively suppressing the default robots.txt block and appending its own:

    User-agent: *
Disallow:

    You may want to ask Yoast SEO support why they chose to do that.

    In any case, the resulting robots.txt file, even with User-agent: GPTBot Disallow: / at the top, should be fine. Are you getting errors in your Google Search Console under Settings > robots.txt?

Viewing 1 replies (of 1 total)
  • The topic ‘Sitemap generated without default WordPress elements’ is closed to new replies.