Blocked by robots, HTML parse error

  • Resolved aamerkheraj

    (@aamerkheraj)


    8 years ago

    Hi

    I’m getting the following errors but I don’t know how to fix them. Could you please help?
    ========================================================================================
    Custom Status Codes

    The Link Checker uses the following custom status codes:

    598 – Blocked by robots: The Link Checker was not able to access the page because the access was blocked by the robots exclusion protocol.

    599 – HTML parse error: The HTML code of this page could not be parsed because of an error in the code or because the page was larger than 50 MB.

    Please note that it is also possible that a website returns these status codes and if this is the case, they probably have another meaning.
    ========================================================================================

    Blocked by robots – What can I do to fix this?

    HTML parse error – I don’t see anything wrong with the HTML code. I do have some parameters like target=_blank but that should cause errors.

    Thanks,
    Aamer

Viewing 5 replies - 1 through 5 (of 5 total)
  • Plugin Author Marco Beierer

    (@mbsec)

    Hi Aamer

    Thank you for using the Link Checker.

    The “blocked by robots” error means that the Link Checker is blocked in your robots.txt file. This file is normally placed in the root directory of your site. If you have never heard about the robots.txt, you could have a look at the Wikipedia article.

    To solve the issue, you could allow the Link Checker to visit all pages by adding the following two lines at the end of your robots.txt. Just make sure that an empty line is between these lines and the current last line.

    User-agent: MB-LinkChecker
Disallow:

    For the “HTML parse error” I would need to have a closer look. Could you send me the link to the affected page?

    Best regards,
    Marco

    Thread Starter aamerkheraj

    (@aamerkheraj)

    I will look into the robots.txt file.

    HTML parse error – I don’t know which pages are affected. Is there any way to check?

    Plugin Author Marco Beierer

    (@mbsec)

    Normally the affected pages are shown in the results table.

    Or do you just think you got these errors because they are shown in the “custom status codes” section? If so, it may be a misunderstanding, because this section is just for documentation and should help with the interpretation of the results. It does not mean that you are affected by these errors.

    Have you already started the Link Checker once?

    Thread Starter aamerkheraj

    (@aamerkheraj)

    Makes sense. I was just reading the documentation.

    Thanks for correcting me.

    Regards,
    Aamer

    Plugin Author Marco Beierer

    (@mbsec)

    Thank you for your reply.

    I may improve the description of this section in one of the next versions to make this clearer.

    Let me know if you have further questions or something does not work.

    Best regards,
    Marco

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Blocked by robots, HTML parse error’ is closed to new replies.