Block URLs with Rule
-
Hi, if I post in the wrong forum please let me know,
Here is my issue. my website gets a lot of hits from search engine with the URL like this:
140.205.69.138 tripwinego.com – [25/Oct/2022:21:26:14 +0000] “GET /cn/category/news-opinion-analysis/?_page=2/page/12//page/6//page/3//page/4//page/13//page/3//page/5//page/5//page/4//page/19/ HTTP/1.1” 200 29283 “-” “YisouSpider” | TLSv1.2 | 0.966 0.971 0.972 MISS 0 NC:000000 UP:SKIP_CACHE_SET_COOKIE
Obviously, the URL is wrong, but it consumes a lot of resources on the server, and I try to avoid that. My questions are:
1. Does anyone knows why the search engine keep crawling URLs that don’t exist (?_page=2/page/12//page/6//page/3//page/4//page/13//page/3//page/5//page/5//page/4//page/19)
2. Can I block these URLs? Currently, I am using Cloudflare to block these URLs, it works but I would prefer not to use another external service.Thanks for your help.
- This topic was modified 2 years, 1 month ago by . Reason: Moved to Fixing WordPress, this is not an Developing with WordPress topic
The page I need help with: [log in to see the link]
- The topic ‘Block URLs with Rule’ is closed to new replies.