Robots.txt removal problems – can't figure this one out

  • Resolved Henrik

    (@havanajim)


    12 years, 10 months ago

    Hi..am a little confused since I ticked the box to allow search engines to now crawl my site (previously asked for them not to) but Google webmaster tools is telling me I still have robots.txt blocking them so am unable to submit the sitemap.

    Any ideas please?

Viewing 3 replies - 1 through 3 (of 3 total)
  • Thread Starter Henrik

    (@havanajim)

    UPDATE:

    Google has now accepted 5 pages out of the 7 pages in the site map but I don’t see my homepage indexed with it’s meta tag so something seems to be wrong? (My site is https://containerforsale.co.uk)

    I followed the suggestion to write the robots.txt the following way:

    User-agent: *

    Disallow:

    Sitemap: https://containerforsale.co.uk/sitemap.xml

    But have since had a reply post on Google webmaster forum suggesting that the above robots.txt file is invalid and may cause Google not to crawl the site. Is this correct? If so how should it be written please?

    Thanks for any help.

    Sitemap: https://containerforsale.co.uk/sitemap.xml

User-agent: *
Allow: /
    Thread Starter Henrik

    (@havanajim)

    Adpawl, thanks for your reply..

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Robots.txt removal problems – can't figure this one out’ is closed to new replies.