• Hi folks,

    I converted my old site (hosted on github) to WordPress mid-year 2018. It had been indexed regularly up to the switch to WordPress, but has not been crawled/indexed into archive.org since. I suspect a problem with robots.txt:

    Old robots.txt file

    User-Agent: *
    Allow: /.
    Allow: /
    Disallow: /oldblog

    Current robots.txt file

    # Stolen from Yoast page https://yoast.com/robots.txt
    User-Agent: *
    Disallow: /suggest/?*

    Could this be caused by a faulty robots.txt file? Any other effects that could cause this? Many thanks!

Viewing 2 replies - 1 through 2 (of 2 total)
Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Site disappeared from archive.org?’ is closed to new replies.