[Plugin: WordPress SEO by Yoast] Google Webmaster Tools cannot parse sitemap.xml

Add-on bit just discovered: If I correct the post-sitemap.xml by resubmitting it in Webmaster Tools, it tests fine. I then move on to the next xml: page-sitemap.xml, and do the same correction. Tools then reports that sitemap as having no errors; however, the first sitemap (post-sitemap.xml) returns to reporting the same original error that the robots.txt file is blocking the directory mentioned in my original post above.
How curious. Both sitemaps use or refer to the same robots.txt file, and that file has the offensive disallow clause removed, yet the two sitemaps conflict causing the other to have a fit whereby Google Tools reports it as having errors.
And why do I need nine sitemap files???
Anyone? I’m open to ideas.

And can anyone point me to the documentation in the WP SEO plugin that possibly addressed this potential issue so the user can avoid hours of research?

Curious: Am I just suppose to list only one sitemap file in Google Tools for the site; and if so, and let’s say I list the pages-sitemap.xml, does that only list the pages and then all my posts, categories, et al are unknown to google’s search?

… Really would have been nice to have had some documentation….

If I go back to a single sitemap.xml, which is what google originally saw before we started using WP SEO 1.2.3, Tools give us the error: “We were unable to read your Sitemap. It may contain an entry we are unable to recognize. Please validate your Sitemap before resubmitting.”
Well, no wonder! We no longer have a sitemap.xml in the root. How did that happen?

I am experiencing what I believe is the same problem. Using latest wordpress version and latest WPSEO. Google webmaster is telling me that it cant parse my sitemap_index.xml because ‘URL restricted by robots.txt’

robots.txt is generated by wp, no? when I go to my https://site/robots.txt file, I see:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

… which looks ok to my untrained eye. Its not allowing robots to crawl anything under the admin and include directories, right? So anything under the root of the site, including the xml file, should be accessible, no?

thanks,

It’s my understanding that WordPress creeates the original robots.txt file, and that various add-ons such as themes, plugins, etc. either modify it, concatenate it’s content or simply overwrite it.
Correct. What you indicated as being in the robots.txt file would stop search engines from crawling those directories; though that seems like a ‘too-short’ of a list. I would not want them crawling:
/wp-includes, /wp-content/plugins, /error_log, /cgi-bin, /wp-content/themes, /wp-content/install.php, /wp-admin, /wp-content/uploads, and /.htaccess. Your site might need more depending on what else you have installed.
And yes, if it’s not on the disallow list, I would say it’s open for the world to see.
Moving forward, I’m not seeing anyway to determine what the WPSEO plugin has changed or added; and, what it’s requirements are and why are they requirements. There simply isn’t any documentation I can find that explains it. Seems the best or closest I can find is to scour the forums and other discussion areas and hope someone else has already had the same problem. Even then, the bugfixes/revisions are coming out so frequently, it’s tough trying to decipher whether a previously reported issue is from a current version or a previous one.
(I’ve spent 20+ years in IT [hardware, software, network and database administration] and I’ve never seen such ‘shoot from the hip’ methodology. I find it irresponsible. Even a simple set-up of bugzilla would save a lot of potential customers a lot of frustration.

Richard, good point about not wanting search engines to crawl those other directories you mentioned.

I believe my robots.txt is the default that either wp generated, or that wp generated and the WPSEO plugin modified. In either case, its not allowing access to my sitemap (which the plugin also generated) according to google webmaster… so i dont understand at all what is going on.

I didnt have this problem with previous versions of WP in combo with this plugin… and I’m assuming this is all related to the issues you bring up in this thread and what you are experiencing… am I off?

Just as reference to robots.txt issues, here is one article from the author himself from earlier this year:
https://yoast.com/example-robots-txt-wordpress/

In my opinion, …. a typical article from him; and more importantly, I’m sure it makes total sense to him, but apparently Google sees it otherwise and it is Google that determines my rankings and analytics.
Thousands of the best in the world vs the opinion of one.
I spent the day reviewing Google’s approach, and I’m thinking that using their method as stated on this page ( https://support.google.com/analytics/bin/answer.py?hl=en&answer=1009686&topic=1009685&parent=1726910&rd=1 ) is an alternative. I think I’ll set up one site according to Google’s method, run it for a couple of days and see what the raw logs and what Google Analytics look like.
I’m frustrated, and since there are 71 pages (as of this morning) of questions and problem postings in this forum alone for this plugin, I don’t think I’m alone.

In Webmastertool erscheint immer wieder ein Fehler mit folgendem Wortlaut: yoast-ga/outbound-article/https://www.zeitungsgenerator.info. Wie kann ich diesen Fehler beheben. Hat das etwas mit den Einstellungen zu tun. Dank denen die mir weiterhelfen.

These are English language forums. Please use English.

I knew I should of taken more languages in college….

hi just in case this helps anyone…
Just had the same problem (installed yoast, submitted sitemap to google webmaster, had error message “URL restricted by robots.txt”).
Took ages to figure out… realised that I’d installed wordpress in it’s own directory and so my robots.txt said this:

User-Agent: *
Disallow: /wp-content/plugins/

when it should have said this:

User-Agent: *
Disallow: /wordpress/wp-content/plugins/

Then i re-submitted my sitemap_index.xml in google webmaster and I have no errors…

I have had the same problem. Just sick of waiting so I turned off XML stie maps from Yoast and use Google sitemap generator. Works fine. Actually, I like it better, as you are able to add pages to the sitemap manually (when they are created outside of WordPress)

Anyhow, that is my take on this.

satguy01,

Thanks for confirming. I’ve done the same thing, and seems to be working fine here.

We know the guidelines that the plugin suggests (which appear to be no different than many others); and they are part of our normal practices; which are also part of Google’s suggestions. We continue to take the time to follow Google closer and use their suggestions. We do the same for Bing.

afshinmokhtari,
I don’t think you are probably ‘off’ if the access to the robots.txt is inaccessible after the install. If it was my site, I would be concerned as well.

Maybe using the Google Sitemap XML plugin may correct the issue. Let us know if you try it and get any positive changes in your rankings.