Viewing 3 replies - 16 through 18 (of 18 total)
  • Thread Starter internationils

    (@internationils)

    https://htaccess.madewithlove.be/
    https://martinmelin.se/rewrite-rule-tester/
    …tried both of those, no luck. I’d guess it’s a user agent issue maybe, as the site is behind cloudflare? Or maybe another plugin? Other sites I have work fine… I’ll close this issue, I can’t really experiment with it as it’s a live site.
    Thanks for your help…

    alexiscollin

    (@alexiscollin)

    The problem is in .htaccess file:

    # Ban Search Engines, Crawlers to your administrative panel
    # No reasons to access from bots
    # Ultimately Better than the useless robots.txt
    # Did google respect robots.txt?
    # Try google: intitle:phpMyAdmin intext:"Welcome to phpMyAdmin *.*.*" intext:"Log in" -wiki -forum -forums -questions intext:"Cookies must be enabled"

    You can comment the 2 lines under :

    # RewriteCond %{HTTP_USER_AGENT} ^.*(AdsBot-Google|ia_archiver|Scooter|Ask.Jeeves|Baiduspider|Exabot|FAST.Enterprise.Crawler|FAST-WebCrawler|www\.neomo\.de|Gigabot|Mediapartners-Google|Google.Desktop|Feedfetcher-Google|Googlebot|heise-IT-Markt-Crawler|heritrix|ibm.com\cs/crawler|ICCrawler|ichiro|MJ12bot|MetagerBot|msnbot-NewsBlogs|msnbot|msnbot-media|NG-Search|lucene.apache.org|NutchCVS|OmniExplorer_Bot|online.link.validator|psbot0|Seekbot|Sensis.Web.Crawler|SEO.search.Crawler|Seoma.\[SEO.Crawler\]|SEOsearch|Snappy|www.urltrends.com|www.tkl.iis.u-tokyo.ac.jp/~crawler|SynooBot|[email protected]|TurnitinBot|voyager|W3.SiteSearch.Crawler|W3C-checklink|W3C_Validator|www.WISEnutbot.com|yacybot|Yahoo-MMCrawler|Yahoo\!.DE.Slurp|Yahoo\!.Slurp|YahooSeeker).* [NC]
    # RewriteRule .* - [F]

    with hashtag.

    alexiscollin

    (@alexiscollin)

    The problem is in .htaccess file:

    # Ban Search Engines, Crawlers to your administrative panel
    # No reasons to access from bots
    # Ultimately Better than the useless robots.txt
    # Did google respect robots.txt?
    # Try google: intitle:phpMyAdmin intext:"Welcome to phpMyAdmin *.*.*" intext:"Log in" -wiki -forum -forums -questions intext:"Cookies must be enabled"

    You can comment the 2 lines under with hashtag.

Viewing 3 replies - 16 through 18 (of 18 total)
  • The topic ‘robots.txt and sitemap.xml blocked’ is closed to new replies.