Robots…

here is just one of many examples…

48261 66.249.72.45 2006-01-21 12:42:54 /2005/11/11/amman-bombs-kill-p roducer/?p=108 unknown Mozilla/5.0 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)

48260 66.249.72.45 2006-01-21 12:42:50 /2005/11/11/amman-bombs-kill-p roducer/?p=106 unknown Mozilla/5.0 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)

48259 66.249.72.45 2006-01-21 12:42:44 /2005/11/11/amman-bombs-kill-p roducer/?p=110 unknown Mozilla/5.0 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)

48258 66.249.72.45 2006-01-21 12:42:42 /2005/11/11/amman-bombs-kill-p roducer/?p=107 unknown Mozilla/5.0 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)

48256 66.249.72.45 2006-01-21 12:42:37 /2005/11/11/amman-bombs-kill-p roducer/?p=109 unknown Mozilla/5.0 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)

what kind of user agents are these?

wyylhbdhdklh wry adgy nwKhyiu
juusdpoc6qikuaowWdwit6
gmwuktqwnwidgu wtykq1pjfbtibojlfwkm
oFmtg yjtqwbjanaftqt7ttjnyxprbln

i’ve been getting them randomly and they too are going to pages that don’t exist…

is there a way to stop them?

thank you

welcome to the Internet.

lol…i know… just makes ya a lil nervouse… i started lookin around my site at all the pages and shit to see if anything was hacked or messed up…

does anyone know what bot or anything leaves the useragent unknown? I was just hit over 100 times in less than a minute!!! thats a record for me…

is there a way for me to stop that? or make it so that someone has to have a useragent or something?

is anyone else having this problem too?

That’s been happening to me, too, for several months now. My bandwidth has increased tremendously to almost 1GB a month. Over half of that are the bots. Don’t know why but I have started reading a forum specifically on Google. I forget the link now. It might have to do with a new way Google crawls.

Luckily, I have the flexibility to increase my bandwidth to whatever I need.

i don’t think its google. google is pretty good about how they crawl and they always start with the robot.txt file, this one doesn’t. the one(s) with the useragent of unknown will go to the same page (ones that exist and ones that don’t) seven or eight times in a row before moving on and doing the same on a different page.

i have also been getting the following crawler coming from this website…not cool…

FAST Enterprise Crawler 6 used by Schibsted Sok ([email protected])

<meta http-equiv=”Content-Type” content=”text/htmlhttps://mykahil.com/wp-admin/theme-editor.php?file=wp-content/themes/wp-andreas09/header.php&theme=WP-Andreas09&a=te; charset=UTF-8″ />

Just wonderin man…whats all that crap in your meta tags?
I just sim spidered your site so don’t worry…the new one was me ??

I have never seen that line in a meta lookin like that…your also missing like 7 other lines.

I’m wondering if the problem isn’t with your set-up, but with a set-up thats linking to yours and the bots are just crawlin their way ta yours.

that line in mine looks like this….
<META HTTP-EQUIV=”Content-Type” CONTENT=”text/html; charset=ISO-8859-1″>

And mine are set up perfectly…or so I though ??

Also i think that vkaryl should be forced ta change her name to something more “girly” ?? lol

don’t know where that came from… i’ve never edited that part…

The robots read those eh and I don’t think that it is right…you should ditch it in a hurry.

also add these dude….

<META HTTP-EQUIV=”EXPIRES” CONTENT=”0″>
<META NAME=”RESOURCE-TYPE” CONTENT=”DOCUMENT”>
<META NAME=”DISTRIBUTION” CONTENT=”GLOBAL”>
<META NAME=”DESCRIPTION” CONTENT=”cool little discription here”>
<META NAME=”ROBOTS” CONTENT=”ALL”>
<META NAME=”REVISIT-AFTER” CONTENT=”3 DAYS”>
<META NAME=”RATING” CONTENT=”GENERAL”>

and replace the crazy tag with just this part of yours…
<META HTTP-EQUIV=”Content-Type” CONTENT=”text/html;charset=UTF-8″>

Then see what happends….who knows ??

ok… *confused*

what is this supposed to do? cause i’m not going to just add something without knowing what it does. I don’t want robots constantly hitting my site, especially the ones who won’t obey my robot.txt file. the only bot that doesn’t bother me is google’s.

man..that would be alota typin ??

the stuff is safe for sure man…veiw my code source…I’m usein it.

the little discription thing is something you realy should have…right now ya don’t have one.

Also…the revisit after 3 days…change it to 30 days if ya want.

I don’t know if this will fix even a part of your problem, but it might…at the very least the spiders will like ya more for givin them some help and it won’t hurt ya.

I just though id stick in some of the only input i can give ya man..i do know that meta tags can be very powerfull and you havin that one screwed up one could be really bad.

and you can stick what ever ya want in a robots.txt file…they don’t have to read it if they are programed not to.