Robots…

that one of mine is fixed, don’t know how that happened…

as for the others, i’m not really out to help the bots cause they really aren’t helping me whatsoever. but i still need to know more about these things before i’m going to use them. do you have a link to a page that can explain them in detail?

yeah, i know that they don’t have to read my robot.txt file, but from my research and experience, the only ones that don’t are the bad ones…

https://www.meta-tag.net/descriptions.php

has doing this kept those new bots away? the ones who don’t have a useragent or the ones who’s useragent is just a random series of letters and numbers? those seem to be the ones that I and many others are having problems with lately…

I have never had a problem with that and I have like 6 web sites of my own and manage 1 other. Every robot that doesn’t ID it self properly hits my .txt file then leaves.

It maybe some thing to do with the whole blog scene…so many of ya link to each other and the links get crawled then crawled again like in a big loop.

I have noticed that since I set-up my WP like 2 weeks ago I have been useing more bandwidth than I should be and I just submited it to the engines like 2-3 days ago and don’t even have all that much content in it yet.

you can do up your robot.txt file to have a list of alowable indexers and to disalow any others, but it may not totaly work…you seem to be leary of installing peoples stuff right off the board here so maybe you should just google “indexer types” or “indexer names” and give that a go.

this is the robots.txt file im usein in my WP right now…may change it soon cuase I just sorta through this in…im workin on a million htings at once ??

The internet is heaven for the attention deficit dissorderly

User-agent: Mediapartners-Google*
Disallow:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-config.php
Disallow: /wp-mail.php
Disallow: /wp-pass.php
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /functions.php

oh man…triple posting here…sorry ??

this could help ya or at least be interesting ??

https://www.searchengineworld.com/spiders/spider_ips.htm

hey chilly,

I added those meta’s to my header, but the one bot i do allow to crawl me, google, is still doing it everyday. i was under the impression that that one meta told it to only come back every three days… thats what it said on that site you gave the link to as well… hmmm…

Some will do as their told and some will not…i get hit everyday by the same spiders in every site…they don’t seem ta care about that line…unless its becuase the tags are not being placed on every page.

I just got hit by someones personal indexer that they’ve set up on their own…kinda has me worried a bit.

I have noticed and I’ve just recently posted in another thread that out of all of my sites my WP is getting hit the most even though its only been up for a few weeks…my soccer set up gets hit by google 12 times in a day…my WP is getting over 300 hits by google alone.

My bet is that they’ve redone the bots recently to crawl blogs more often due to there fast changing content. If i took out the link i have in this WP set up…my hits would drop off I bet…till someone on here adds a link to mine on theirs..then it would be back up on the hits.

yeah, that happens to me a lot. the one that i seem to be having the most problem with is one that does not have a useragent. it comes up as ‘unknown’ and hits me a couple hundred times within just two minutes or less and it will go to pages/links that just do not exist on my site.

Your issues may have been resolved, I don’t know, so here’s hoping this benefits others:

“the one that i seem to be having the most problem with is one that does not have a useragent.”

Bad Behaviour does block requests from user agents without a name.

“i’m not going to switch hosting plans like it suggests, i’m getting a way too good of a deal for what i got.”

Really? From what you’re complaining about, it certainly doesn’t seem like it. There are many documented issues with godaddy.com’s service. Like with everything else, you certainly get what you pay for.

Well, first, Bad Behavior does not work on a GoDaddy server.

Second, if you can find me a better deal where I get the same or more for less money, then by all means share with us all.

I have pretty much solved the bot problem that I was having, hence me not posting in here in over a month now… I bypassed GoDaddy’s lack of support by use of my htaccess file. with a few mod_rewite additions the bots have been taken care of…

Yeah, GoDaddy doesn’t have the best service, I wouldn’t recomend it for someone just starting out, but once you learn how to do more advanced thing on your own, it isn’t all that bad.

Thank you,
Kahil

“Well, first, Bad Behavior does not work on a GoDaddy server.”

I’ve re-read the posts (and I must have missed the explanation) but just out of curiousity, why not?

“Second, if you can find me a better deal where I get the same or more for less money, then by all means share with us all.”

Going by your definition of a “good deal”, which seems to mainly focus on the money you’re spending: No, I can’t. I’m with a provider catering for businesses. A lot more than godaddy in price but also in features, functionality and support which I value very highly.

bad behavior doesn’t work with GoDaddy because of someothing that their servers do. I can’t remember where I read it exactly, but its something like shadowing. So that when you visit your own site, bad behavior thinks that you are some bot ro something…

Yeah, if I were to pick a service for a business, GoDaddy would not be my choice, but for just a blog or simple site, I say why not…

I do recall BB blocking you in the WP admin interface when the function wp_get_http_headers() is called but WP fixed that with V2 by supplying a user agent string when issuing a HTTP “HEAD” request:

WP 1.5.2: $head = "HEAD $file HTTP/1.1rnHOST: $hostrnrn";

WP 2.0.2: $head = "HEAD $file HTTP/1.1rnHOST: $hostrnUser-Agent: WordPress/" . $wp_version . "rnrn";

BB would then intercept the request with a “A User-Agent is required but none was provided” error.

I don’t recall ever hearing or reading anyone recommend godaddy for shared hosting services, largely due to their technical staff being a pack of incompetent idiots, but still, they are ridiculously cheap, aren’t they?

EDIT: I’ve done some research and I see what you mean by why godaddy are to blame by mangling HTTP headers for starters:

https://error.wordpress.com/2006/01/01/godaddy-sucks/

yeah…

through my htaccess i’ve taken care of all the bad bots that were crawling… the only thing i’m still fighting is foreign IPs, mostly european on the RIPE network, attacking my site by trying to go to my guestbook, which doesn’t exist. The problem is that they are coming from many many different IP addresses. I have almost 100 different ones that i’m blocking through my htaccess file and that number grows each day… I don’t think that BB would even block them all cause they format the URI that they are trying to go to so that it looks like a search…