Robots won't crawl my site

  • James

    (@fargocellcycle)


    11 years, 4 months ago

    [ Moderator note: please do not put your URL in the topic title. ]

    Okay so I am somewhat dumb when it comes to website stuff.

    I am in a predicament. I hired a guy to design a website for me and before he finished it POOF and he is gone.

    I have been slowly teaching myself how to make changes through stuff like youtube and google but I am totally stumped on this one.

    I have gone to https://www.icarefargo.com/robots.txt and my WP site is disallowed on every front. I have searched in google, yahoo & bing my sites url and they all indicate that no description is found/ allowed due to site robots.

    I have read the countless posts regarding the robots file being embedded in the root directory. I have installed plug ins and selected for the bots to crawl my site but no success. I have even added some SEO plug ins and added descriptions etc for my pages but it is still registering that bots are disallowing crawling.

    I am at a dead end and don’t feel like I will ever get the bots to successfully crawl my site which is not a good thing as I have all sorts of content there that I think would start to index my site above my competition.

    I hope that someone has som direction for me. I have even searched the .php files and found NOTHING pertaining to any robots text or anything else.

    So the least I can say is….. HELPPPPPPPPPPP!

Viewing 2 replies - 1 through 2 (of 2 total)

  • nice site –

    I answer as we are in the same type of business

    your robots.txt file is in the public_html directory on your hosting server

    you need to change this file to remove the disallow

    your designer may have put the disallow in the file during the design and launch process to make sure the site was not indexed while under development – hard to say what the story might be but this is a rather definitive way to make sure google does not see or index your site

    to make the changes to the file you can go to your hosting provider or the provider usually has tools that you have access to to edit files on the server

    you also might use FileZilla to access files on your hosting server

    at a very simple level your robots.txt file needs to look something like this:

    User-Agent: *?
    Disallow: /wp-content/plugins/

    the above disallows wp-content plugins

    you may chose to disallow other files or types of files you do not want google to index

    do you know about google webmaster tools?

    are you doing anything about SEO?

    sorry didn’t do the link correctly
    my site is iRedCord.com

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Robots won't crawl my site’ is closed to new replies.