My own shared server is crawling my site? Why?
-
My own shared server is crawling my site? Why?
Lately I’ve been policing my access logs more closely to try to systematically weed out those crawlers and bots that do me no good, and there among them crawling my site I find the very shared server my site is hosted on. Typically a bunch of GET requests for both new and old pages interspersed with POSTs for cron jobs.
At first I thought it might be my Google XML Sitemaps plugin updating my sitemap daily, but after deactivating it the log entries keep appearing. Unfortunately, my logs don’t go back far enough for me to tell when this might have begun.
I can’t imagine how any syntax in my .htaccess or wp-config files might be generating such complex and sophisticated autonomous behavior, and so far I’ve found no reference to anything similar online. A Dreamhost support tech mentioned in passing that it appeared to be the recursive behavior of a script, but since I only have a standard WP installation with up-to-date plugins I have no way of imagining what that might mean.
Any insights into what this is and how innocuous or pathological it might be are appreciated.
- The topic ‘My own shared server is crawling my site? Why?’ is closed to new replies.