Allow Heritrix crawler

  • Resolved Sjors

    (@sjors-roelofs)


    2 years ago

    Hi!

    I have a site that needs to be crawled once a day. The crawler for this site is Heritrix. Currently, the crawl fails because iThemes Security blocks the Heritrix user agent by a line in the htaccess file.

    If I remove this line, it gets automatically added again after a while.

    Is there a way to always allow this crawler/user-agent to crawl the website, without completely disabling iThemes Security? I cannot find any settings within the plugin settings to allow this, only to block certain user agents.

    Thanks!

Viewing 2 replies - 1 through 2 (of 2 total)
  • nlpro

    (@nlpro)

    Hee Sjors,

    Navigate to:

    Security > Settings > CONFIGURE > Lockouts > Ban Users

    and disable the Default Ban List setting.

    +++++ To prevent any confusion, I’m not iThemes +++++

    Plugin Support chandelierrr

    (@shanedelierrr)

    Hi @sjors-roelofs, I hope the information provided by nlpro helped resolve the issue. Since we haven’t received a response, I’ll mark this post as resolved. If you still need some assistance, feel free to open a new support topic, and we’d be happy to assist. Thank you!

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Allow Heritrix crawler’ is closed to new replies.