Google can't crawl my website

  • Hi people!
    I hope you can help me with this one. Recently I took over a website from another developer and everything’s great, except for no SEO is done. I wanted to set pup all the tags and title and stuff, and found this in the process.
    “A description for this result is not available because of this site’s robots.txt – learn more.”
    So I started researching the robot.txt thing and came to understand, that I do not have a physical one in my root folder.

    The website runs on a Brooklyn child Theme by United Themes. The web host they use is Bluehost.

    Things I tried.Creating and empty document robots.txt and placing it in root folder. Checking admin panel for privacy settings. Everything seems in order. I tried seeing if there’s any sort of blocker on BlueHost, but everything was alright there as well (as much as I can tell). Then I tried Google webmaster tools. I created a sitemap that I linked. Then I tried to fetch robots.txt as google. It fetched the file and said all is good. It got no crawl errors or site errors.

    Here’s the site: bbg-films.com
    here’s the robot big-films.com/robots.txt

    Anything that I’ve done still doesn’t show any results on google.
    It only indexes it If I google bbg-films.com, and then in the description it says A “description for this result is not available because of this site’s robots.txt – learn more.”
    If I search bbg-films.com on Bing it shows up.

    What Should I do?

Viewing 4 replies - 1 through 4 (of 4 total)

  • Hi

    Have you checked your .htaccess file to make sure they are no errors blocking access?

    What response have you got from Google webmaster tools in response to your sitemap?

    Have you also made sure the option in your WordPress settings ‘discourage search engines from indexing this site’ is not checked?

    Thanks
    Martin

    Thread Starter matisszdanovs

    (@matisszdanovs)

    Hi Martin!

    The .htaccess has this inside it
    “# Use PHP5.4 as default
    AddHandler application/x-httpd-php54 .php

    # BEGIN WordPress
    <IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /~bbgfilms/
    RewriteRule ^index\.php$ – [L]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . /~bbgfilms/index.php [L]
    </IfModule>

    # END WordPress”

    Honest to god I have no clue how to check if there are errors blocking access.

    Webmaster says my sitemap is still pending.

    Yes I’ve made sure that ‘discourage search engines’ is not checked.

    Thanks

    Matiss

    Everything certainly looks normal.

    Search engines can take around 2 weeks to crawl a site, have you check other sites like bing? yahoo? etc. Are they listing your site?

    I’m seeing a wonky canonical link url (located in <head>) on bbg-films.com. Canonical links act as “soft” 301 redirects. You basically tell SE’s that bbg-films.com redirects to https://69.195.124.217/~bbgfilms/, latter is blocked via robots.txt and this is hindering your site from indexing and ranking on Google.

    Hope this helps!

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Google can't crawl my website’ is closed to new replies.