Site Map No Index issues

  • Resolved nubiesan

    (@nubiesan)


    9 months, 2 weeks ago

    I have had major issues with my robots.txt file, which I managed to fix, but the sitemap fix is beyond me.

    Google cannot fetch the site map, since it says there is a noindex directive in the X-Robots-Tag?. I found the file that contains the directive, but I have no idea how to edit it. This is what curl found:

    Server: nginx
    Date: Sun, 18 Feb 2024 19:45:09 GMT
    Content-Type: text/xml; charset=UTF-8
    Transfer-Encoding: chunked
    Connection: keep-alive
    Vary: Accept-Encoding
    X-Robots-Tag: noindex, follow
    X-Httpd-Modphp: 1
    X-CDN-C: static
    Content-Encoding: gzip
    X-SG-CDN: 1
    X-Proxy-Cache: MISS

    How do I get rid of this? At this point I would like to disable the AIOSEO sitemaps and use the wordpress generated maps. My site is not being crawled by google, which is causing lost sales. Any ideas?

    The page I need help with: [log in to see the link]

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Author arnaudbroes

    (@arnaudbroes)

    Hey @nubiesan,

    I’m sorry to hear that you had some issues with your robots.txt file.

    The noindex directive for the sitemap is meant to prevent search engines from indexing your sitemap and including it in their search results, but it should not prevent them from fetching and parsing it.

    Can you let us know where you are seeing that Google cannot your sitemap due to a noindex directive (perhaps add a screenshot)? That will allow us to assist you better.

    Thread Starter nubiesan

    (@nubiesan)

    I took screen shots of the error message I am getting, as well as a copy of the robots.txt file.

    Today’s message is different than the one I sent yesterday, as it makes no reference to the noindex.

    There are fewer products affected today. I have not changed anything in the settings or descriptions, so I don’t know what’s going on.

    The link is to a document with the 3 screenshots. It’s on dropbox.

    https://www.dropbox.com/scl/fi/hb1cguaijvddc6x5rd9cn/no-scan-screens.docx?rlkey=343h7hqt3hlxopx2gpo15pixo&dl=0

    Plugin Support J Burns

    (@subiewrx)

    Hi @nubiesan,

    Thanks for sharing the screenshots.

    Based on this message – https://a.supportally.com/i/6Yd6wN, you’ll want to remove these lines from your robots.txt – https://a.supportally.com/i/lXylq5

    In checking your robots.txt, it appears that you have a static file on your server. You can delete this file or import it in All in One SEO > Tools > Robots.txt Editor.

    Then toggle OFF the Enable Customer Robots.txt which will remove those rules if they are still present.

    Please let me know if you have any other questions.

    Thanks!

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Site Map No Index issues’ is closed to new replies.