Site disappeared from archive.org?
-
Hi folks,
I converted my old site (hosted on github) to WordPress mid-year 2018. It had been indexed regularly up to the switch to WordPress, but has not been crawled/indexed into archive.org since. I suspect a problem with robots.txt:
Old robots.txt file
User-Agent: * Allow: /. Allow: / Disallow: /oldblog
Current robots.txt file
# Stolen from Yoast page https://yoast.com/robots.txt User-Agent: * Disallow: /suggest/?*
Could this be caused by a faulty robots.txt file? Any other effects that could cause this? Many thanks!
Viewing 2 replies - 1 through 2 (of 2 total)
Viewing 2 replies - 1 through 2 (of 2 total)
- The topic ‘Site disappeared from archive.org?’ is closed to new replies.