Googlebot cannot access CSS and JS files

Question removed because question had already been asked – sorry!

Same here..

According to webmastertools my only blocked source would be this: /wp-admin/js/media-upload.min.js It’s causing to block some pages on my site inclusing the homepage.

Is it normal that this file is inside the wp-admin/?

And I got confused with the solution; should we remove all in robots.txt and leave it blank or do we have to put the list in of disallow and allow?

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/cache/
Allow: /wp-content/themes/
Allow: /wp-content/plugins/
Allow: /wp-content/uploads/
Allow: /wp-includes/css/
Allow: /wp-includes/js/
Allow: /wp-includes/images/

OR:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/cache/

User-agent: Googlebot
Disallow: /wp-admin/

OR:

********************************
******THE SOLUTION******
********************************

We should just keep all our robots.txt files blank from now on. There is no longer a need to block core wp directories.

Thanks!

Whatever I would like to try, I cannot change my robots.txt file?

How did you guys solve this? If I place it in my root directory it still shows the virtual WP file with:

User-Agent: *
Disallow: /wp-admin/

This file is not in my directory.. so where does it come from? i suppose a default from WP, but the question is how can I edit this ??

Google is not the ONLY search engine or robot out there is it? Plenty of malware bots like to dig around in admin and includes folders to see what’s what. I’m not treating this as an emergency

There’s a fairly good chance that malicious bots wouldn’t pay attention to your robots.txt though, which means that keeping those blocks in place purely for that reason would probably be pointless.

I’ve updated and submitted by robots.txt file, which now looks like this:

User-Agent: *
Disallow: /wp-*

User-Agent: Googlebot
Allow: /*.js*
Allow: /*.css*

However, when I fetch and render as Google, .js and .css files are still being blocked…

Any ideas?

Cheers

@stlewi85

my GoogleBot code looks like the below and it works.

#Googlebot
User-agent: Googlebot
Allow: *.css
Allow: *.js

whereas yours has a ‘/’ before allowing .js and .css

perhaps that’s what’s wrong?

Thanks @vimiio, I’ll give that a whirl.

@vimiio – Nope, it hasn’t worked. Here’s what I’m seeing when I fetch and render my homepage – https://i.imgur.com/wmGNGui.png

And here’s my live robots.txt. file – https://i.imgur.com/CeYprBB.png

Could this be because the .CSS and .JS files are from third parties?

Hi @stlewi85

I had the same problem. It seems I had my robots.txt file in the wrong folder!
If you look at your site and add /robots.txt at the end it shows you the actual file it uses. And this file is the default of WordPress.

Just put your robots.txt file inside you theme folder!

ps. I used this one and solved my problem:

User-Agent: *
Disallow: /wp-admin/

User-Agent: Googlebot
Allow: /*.js*
Allow: /*.css*

Your robots.txt is now:

User-agent: *
Disallow: /

@stlewi85 They are external resources? So not controlled by your Robots.txt riles.

@colin mcdermott – I thought as much. I wonder if this will negatively affect the site.

No it shouldn’t, not even 0.01%, you can’t control external resources like that…

If I opened up access to everything in my robots.txt file and I was using an older version of WordPress, would that cause any unexpected problems? I’ve read that WordPress has noindex built in to their admin and includes directories… Just want to make sure I’m not opening an unexpected can of worms

@rhorsley33 sure it would be fine. Though there is no harm disallowing /wp-admin/ – there is nothing there that needs to be indexed…