Robots.txt

  • I created a website for a client. Now the search engines won’t show the description.

    I get this response when I google it:

    A description for this result is not available because of this site’s robots.txt – learn more.

    I’m not sure where to begin. The website is https://ActOutsideTheBox.com.

    A couple of days ago I found and #unfollow command in a social media plugin. I disapled the social media plugin and added an SEO plugin.

    Is it possible that it will take a few weeks to get this cleared up?

    Thanks so much for your help.

Viewing 5 replies - 1 through 5 (of 5 total)

  • https://www.actoutsidethebox.com/robots.txt

    Currently set to 

    User-agent: *
Disallow: /

    Check in Dashboard > Settings > Reading > Search Engine Visibility

    Creating a physical robots.txt will also override the WordPress virtual file.

    Some reference materials: Search Engine Optimization for WordPress

    Thread Starter bittenbyazebra

    (@bittenbyazebra)

    Thanks, Clayton!

    The Discourage Search Engine Visibility is not checked.

    I wish there was a way to edit this file. Hmm.

    I added two plugins, submitted changes and the robots.txt remains the same.

    Does this mean I have create a file and go in via ftp or some other way?

    Thanks so much for you advice and help!

    Bob

    Thread Starter bittenbyazebra

    (@bittenbyazebra)

    I actually went into the site via ftp and changed the robots.txt to

    User-agent: *
    Crawl-delay: 1
    Disallow: /wp-content/plugins/
    Disallow: /wp-admin/

    It doesn’t seem to have taken. Any thoughts?

    I’ve used what i have on my personal site and it works quite well with search bots.

    Bob

    It doesn’t seem to have taken. Any thoughts?

    Visit this link and read the file. https://www.actoutsidethebox.com/robots.txt

    It still says

    User-agent: *
Disallow: /

    I see cloudflare-nginx in the server response for the site. Did you update the CloudFlare cache after making your changes?

    Thread Starter bittenbyazebra

    (@bittenbyazebra)

    I don’t have access to the CloudFlare cache through WordPress. I tried using Wordfence to clear the cache but that didn’t seem to do anything.

    I’ve got an email out to the web hosting people and hopefully they may be able to help. I’ll keep you posted and thanks again for your help!

    Bob

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Robots.txt’ is closed to new replies.