Existing Robots.txt unable to remove or EDIT!!

  • Resolved angsin

    (@angsin)


    3 years, 2 months ago

    I have just installed Smartcrawler plugin to website and removed Yoast. But Smartcrawler cannot edit due to existing robot.txt.

    I have tried removing robots.txt, deactivated yoast and auto-import configuration into SmartCrawl.

    Yet nothing is working, very frustrating, considering deleting SmartCrawler.

Viewing 4 replies - 1 through 4 (of 4 total)
  • Plugin Support Nithin – WPMU DEV Support

    (@wpmudevsupport11)

    Hi @angsin ,

    Sorry to hear about the issue noticed. In general, if there are any physical robots.txt files created, it would override the dynamic robots.txt of SmartCrawl.

    Deleting the robots.txt file as you have stated should have resolved such issues. since we don’t have a link to the website where you are facing the issue, it’s tough to say what might have caused it.

    Have you tried flushing the permalinks to ensure the issue isn’t due to that? You should find the option in the default WP settings, under Settings > Permalinks and make sure “Common Settings” has “Post name” selected, and click the “Save Changes” button.

    The “Save Changes” button should help with flushing the permalinks, so please do check and see whether re-saving the settings help or not.

    If you still face issue even after that, then if possible please do share the website URL, so that we could have a better idea of the issue.

    Looking forward to your response.

    Kind Regards,
    Nithin

    Thread Starter angsin

    (@angsin)

    Permalink is with “post name” already selected

    the website: beautylondon.no

    Hello @angsin !

    Hope you’re doing well!

    I checked the /robots.txt on that domain – unfortunately by the headers themselves I can’t tell whether this is the SmartCrawl robots.txt file or an actual file residing on the server.

    Important to keep in mind what my colleague said earlier – if there is an actual robots.txt file in the root directory of the site, it will be used instead of the SmartCrawl settings. This is because a web server will not go through WordPress code when it detects a file with a matching name already on the filesystem.

    Please make sure that the robots.txt file in the main site’s directory is removed or renamed – usually it’s located in the public_html folder or similar. Once it’s renamed or removed you will be able to edit it via SmartCrawl.

    Kind regards,
    Pawel

    Plugin Support Predrag – WPMU DEV Support

    (@wpmudev-support1)

    Thank you for using SmartCrawl. We are going to close this ticket for now because we have not heard from you in a while. If you still need support we can reopen the thread.

    Cheers,
    Predrag

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Existing Robots.txt unable to remove or EDIT!!’ is closed to new replies.