Googlebot cannot access CSS and JS files

Methinks Google is smart enough to not index wp-admin and wp-includes, unless you tell them too, so this whole discussion may simply be a mute point.

Bottomline, is Google wants to index your CSS and JS and won’t have it any other way. So the final solution is to simply bow down and accept…

The apparent new standard should close this subject:

User-Agent: *
Disallow: /wp-admin/

User-Agent: Googlebot
Allow: /*.js*
Allow: /*.css*

Concur?

I’ve reduced the disallows to nothing and when using the fetch and render tool, get a lovely long list of plugin related CSS and JS files that Google has an issue with – how are we supposed to overcome these?

Hey Guys…

Good thread. Thanks for all the responses.

I mod’d robots.txt and we’re good to go:

User-Agent: *
Disallow: /wp-admin/

User-Agent: Googlebot
Allow: /*.js*
Allow: /*.css*

> I’ve reduced the disallows to nothing and when using the fetch and render tool, get a lovely long list of plugin related CSS and JS files that Google has an issue with

@kesbutters What are the error messages you are getting?

GoogleBot may be smart, but its obviously not smart enough to know that none of the files in wp-admin are necessary to properly index my sites. They’ll have to trust me on that.

If I suggest in my robots.txt that they should not index wp-admin/, then they should not index wp-admin/.
If they absolutely want to index wp-admin/ then there’s nothing stopping them technically – they are welcome to do so.
If I don’t want anyone to index wp-admin/, I would block access.

I doubt this will result in “suboptimal ranking”, because this is affecting nearly all WP-installs, 75 million sites or so.
So I’m not going to touch this for now, and I expect references to files in wp-admin (ref Wanderer’s comment) to be fixed by a next WP update ish.

The following, not on the robots.txt file (although I’ve had the warning msgs) but showing up in the fetch and render tool in Webmaster tools. All plugins up-to-date and there’s a list of around 15-20.

URL
https://www.markitwrite.com/wp-content/plugins/testimonials-widget/includes/libraries/bxslider-4/dist/jquery.bxslider.css?ver=4.2.3

Type Style Sheet Reason Blocked robots.txt Tester

Type is either stylesheet or JS

@kesbutters Your site is markitwrite.com ?

Looks like you are blocking access to your plugins folder as per below:

Disallow: /wp-content/plugins/

I would remove that from your robots.txt – as Google has shown there are files in there that they need to access to properly render your site.

Hi Colin – nopes, I’ve edited the robots.txt file and taken everything out. Even wp-admin I’ve unblocked so am pretty confused as to why these are showing up as they are deffo not in the robots file any longer – I’ve used a WP robots.txt file plugin to edit it however, as it doesn’t appear to be in my root folder when accessing through FTP.

Hi, I am confused, are we not talking about https://www.markitwrite.com/robots.txt ? Here is what is in the file:

User-agent: *
Crawl-delay: 1
Disallow: /wp-content/plugins/
Disallow: /wp-admin/

If not can you specificy your Robots.txt URL?

Hi Colin,

Ok so it looks like the plugin that I used to access and edit the robots.txt file doesn’t work!

So I have to find a way to discover and download the file via FTP and I can’t find it in the root folder.

Will check it out and look through the files properly later – thanks Colin, this was obviously a case of me being a bit dim and not checking the actual robots file link first – I’ll let you know if it does the trick (if I can find the file, that is).

Hi Kesbutters,

See my comment earlier. The file is not in your root, you should add it there yourself and it will overwrite the defailt virtual file of WordPress.

Ps. I placed it in my themes folder and worked perfectly.

Hope this helps!

@kesbutters glad to help! It can be a pain sometimes, I have had that happen before also.

Bizarre coincidence, as soon as I had replied earlier – I went to Tweetdeck and the first tweet I saw in my stream was from you! I already follow you with my business Search Candy. How unbelievable a coincidence is that!

@wanderer – thanks! I will go and do that.

@colin – hahaha! Yes I am pretty prolific on Twitter as I’m sure you’ve noted ?? I see I follow you too so will retweet your blog as a thanks for your help ??

I still can’t believe it! The chance of that must be in the billions. Thanks so much that would be awesome. Have a great day! ??

CAN THIS AFFECT ADSENSE REVENUE?

On July 26 my adsense revenue dropped massively. My CPC’s and CTR’s were considerably lower than usual. (My traffic was normal.) After a few days of this, I figured I was getting a run of low-quality, irrelevant ads and hoped it would return to normal. This morning, 5 days later, the low revenue persisted. I figured I better look into it further.

I thought back to the July 28 (two days after seeing Adsense revenue drop) emails I received from Google about Googlebot not being able to access my js and css. So I did some research today and was helped by this thread and the article at Beachfront mentioned earlier in this thread.

So I went into the sites to remove the offending disallow wp-admin from the robots.txt files. But today, when I checked the robots.txt files in webmaster tools, it was still showing the older version.

So, I added the additional “allow” lines you suggested in the Beachfront article (although in my case I’m not sure I really need them, but can’t hurt!). In fact, those lines are all I have in the robots.txt files, except for a pointer to my site map:

User-Agent: Googlebot
Allow: /*.js*
Allow: /*.css*

Sitemap: https://www.mysite.com/sitemap.xml

* Two of my sites also have a disallow at the beginning of robots.txt of an an old folder unrelated to WP.

Then I deleted the Supercache cache on all my sites, and then purged the cache in Cloudflare. Then in Webmaster Tools I submitted the new robots.txt for each site, and it now shows the current file when I look at it in Webmaster Tools.

At this point my fingers are crossed, waiting to see if there is any immediate change in Adsense revenue. I should know my end of day, or tomorrow — IF Adsense immediately changes behavior when it sees the new robots.txt files.

I have no idea if this actually does affect Adsense, but I have read that if Googlebot has crawl issues it will show lower quality ads. The timing is so coincidental, I’m hoping that the robots.txt was the problem.

Hs anyone else noticed Adsense CTR and CPC drops this week?