General HTTP Error

Hello @hardseed

Sorry to hear about your sitemap. We have some questions:

1. Can you please confirm if you are on the latest Yoast SEO 9.6?
2. Can you share your sitemap URL so we can check?
3. Can you share a screenshot of the error?

This issue was discovered a few days ago when I found that one page in particular was not being indexed: https://greentecct.com/block-molds/

Answers as follows:
– Using Yoast 9.6
– Sitemap: https://greentecct.com/sitemap_index.xml
– Cannot attach a screenshot but this is what I just did and got:
Entered the post sitemap URL into Add a new sitemap in Google search console. Got last read 2/4/19, 14 discovered URLs, Sitemap can be read but has errors, General HTTP error, 1 instance, We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.

Thank you for any help you can provide.

Hi hardseed,

Thank you for getting back to us.

Please try deleting the posts sitemap, and submit the sitemap index instead. We have a step-by-step guide here: https://kb.yoast.com/kb/submit-sitemap-search-engines/#google

You don’t need to submit individual sitemaps (such as posts or pages) because the sitemap index includes all your links.

After resubmitting the sitemap index, you’ll need to wait for Google to recrawl your site.

Please let us know if the issue is not resolved.

I deleted all previous and submitted only the index. It is pending with four errors and one warning due to HTTP status errors. 30 pages were submitted with only 7 indexed. This does not seem to be working. Please advise.

Hi,

We would like you to please do the following:

1. Confirm that your site in Google Search Console is exactly: https://greentecct.com. It needs to have the HTTPs as part of it.

2. We suggest having the default rules for your robots.txt file: https://greentecct.com/robots.txt. This guide explains how to edit the robots.txt:?https://kb.yoast.com/kb/how-to-edit-robots-txt-through-yoast-seo/.

You can use the defaults below.

User-agent: * 
Disallow: /wp-admin/ 
Allow: /wp-admin/admin-ajax.php 

3. Clear all your caching from your theme, plugin, server, CDN like CloudFlare or browser. If you are not sure how to clear caching from theme/plugin please speak to those authors. If you want to clear cache from the server, please speak to your host provider. To clear the cache from a browser use this guide:?https://kb.yoast.com/kb/how-to-clear-my-browsers-cache/.

4. Delete your sitemap in Google. Doing so will not hurt the SEO of the site. This guide explains more (scroll to the bottom):?https://support.google.com/webmasters/answer/183669?hl=en.

5. Submit the sitemaps again. This guide explains more:?https://kb.yoast.com/kb/submit-sitemap-search-engines/.

6. Monitor the results to see if the issues resolve.

Note, if you do get errors we ask that you please describe each error exactly as Google says it or take screenshots. You can use a tool like: https://pasteboard.co/ to share screenshots. Please also provide some example URLs that Google is flagging.

Followed your suggestions but the situation remains the same. Here are screenshots:
https://pasteboard.co/I332Wy9.jpg
https://pasteboard.co/I334K5K.jpg
https://pasteboard.co/I3366Sg.jpg
https://pasteboard.co/I336LeS.jpg
https://pasteboard.co/I338afR.jpg
https://pasteboard.co/I339zZ8.jpg

Looking forward to your quick response.
Thanks!

And if this helps:

https://pasteboard.co/I33hAMv.jpg
https://pasteboard.co/I33iMlA.jpg

We reviewed the sitemap and several pages on your site and found that they are returning soft 404s, including the sitemap itself so the issue is not specific to the sitemap:

cbailey$ curl -I https://greentecct.com/block-molds/
HTTP/1.1 404 Not Found
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Type: text/html; charset=UTF-8
X-Port: port_10934
X-Cacheable: YES:Forced
Date: Wed, 27 Feb 2019 17:39:40 GMT
Age: 3
Vary: Accept-Encoding, User-Agent
X-Cache: uncached
X-Cache-Hit: MISS
X-Backend: all_requests
Connection: keep-alive
Via: http/1.1 a2nlwpproxy023.prod.iad2.secureserver.net (ApacheTrafficServer/7.1.2 [uSc s f p eN:tOc  i p sS])
Server: ATS/7.1.2

cbailey$ curl -I https://greentecct.com/about-us/
HTTP/1.1 404 Not Found
X-Pingback: https://greentecct.com/xmlrpc.php
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Type: text/html; charset=UTF-8
X-Port: port_10934
X-Cacheable: YES:Forced
Date: Wed, 27 Feb 2019 18:03:10 GMT
Age: 1
Vary: Accept-Encoding, User-Agent
X-Cache: uncached
X-Cache-Hit: MISS
X-Backend: all_requests
Connection: keep-alive
Via: http/1.1 a2nlwpproxy023.prod.iad2.secureserver.net (ApacheTrafficServer/7.1.2 [uSc s f p eN:tOc  i p sS])
Server: ATS/7.1.2

cbailey$ curl -I https://greentecct.com/panel-molds/
HTTP/1.1 404 Not Found
X-Pingback: https://greentecct.com/xmlrpc.php
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Type: text/html; charset=UTF-8
X-Port: port_10934
X-Cacheable: YES:Forced
Date: Wed, 27 Feb 2019 18:15:19 GMT
Age: 4
Vary: Accept-Encoding, User-Agent
X-Cache: uncached
X-Cache-Hit: MISS
X-Backend: all_requests
Connection: keep-alive
Via: http/1.1 a2nlwpproxy008.prod.iad2.secureserver.net (ApacheTrafficServer/7.1.2 [uSc s f p eN:tOc  i p sS])
Server: ATS/7.1.2

If you think that your page is incorrectly flagged as a soft 404, use the URL Inspection tool to examine the rendered content and the returned HTTP code. If the rendered page is blank, or nearly blank, it could be that your page references many resources that can’t be loaded (images, scripts, and other non-textual elements), which can be interpreted as a soft 404. Reasons that resources can’t be loaded include blocked resources (blocked by robots.txt), having too many resources on a page, or slow loading/very large resources. The URL Inspection tool should list which resources could not be loaded, and also show you the rendered live page.

Do you guys read the previous comments? I previously sent a screenshot of a URL Inspection but here it is again:
https://pasteboard.co/I3cpAeE.jpg
This page is not blocked by robots.txt- I am even using your recommended robots.txt setting! It also does not have any suddenly appearing elements that are difficult to load.
This is what I get when I click on Open Report:
https://pasteboard.co/I3cufJh.jpg
– /estimate-request-2 yields a 404 which it should
– /quote-request yields a quote form which it should
This is what I get with a Live Test:
https://pasteboard.co/I3cuFep.jpg
Note that it is not possible, as you suggest, to see “which resources could not be loaded, and also show you the rendered live page”
Here is the Bing Webmaster crawl info:
https://pasteboard.co/I3cyBcI.jpg
Why is a different person working on this every time, no wonder the communication is so inconsistent? Please bump this up to someone more senior who can help. Reports on the Internet suggest that Yoast is having problems with this plugin. I look forward to your fast response.

Please do the following:
1. Go to https://httpstatus.io/
2. Enter https://greentecct.com/2019/01/press-release/
3. Click ‘Check’

What status code is shown for the page? We see a 404 Not Found status code.

Now do the following:
1. Login to your website admin
2. Go to Admin > Plugins
3. Deactivate Yoast SEO
4. Clear site and server cache, if in use
5. Repeat steps above

If the check returns a 404, Yoast SEO does not cause the 404 Not Found message. Performing a conflict check may help locate what causes the 404 message.

If the check returns a different status code, what code is returned?

Now that was helpful, thank you. Amboutwe, I flushed the cache in Managed WordPress and Cleared Cache for WP Rocket after deactivating Yoast and still got a 404. What conflicts could there be? Could it be the other plugins or Godaddy hosting settings? Here are the current plugins, are there any known conflicts with any of these:
https://pasteboard.co/I3if6LQ.png
https://pasteboard.co/I3ifzKET.png
I look forward to your quick response.

None of those plugins jump out immediately as something that would modify the status code. The next step would be to disable all plugins, switch to a default theme and check the status code again.

Deactivating your plugins or switching themes on a production site may not be ideal. Have you ever considered setting up a staging environment for your tests? This means that you can do all the testing and only change the live site when all works like intended.

Your host might be able to help
Some popular web hosts offer quick setups for a staging site so please contact your web host for assistance. They will probably be able to help you out.

Alternative options to set up a staging environment
If your web host does not offer staging sites, the WordPress plugin repository offers staging plugins or you can create your own staging site.

Amboutwe,

My Godaddy account has staging, however, when I tried to copy Production to Staging only the Core was copied, Plugins and Theme had copy errors. I have been having Plugin issues with Godaddy for some weeks now which is why I now suspect that they are behind my sitemap issues. I tried to use the WP Staging plugin to create a staging site but got an Installation Failed: -1 error which has been a common issue with Godaddy of late. Any comments?

Regards,
Neil

Unfortunately, we are unable to confirm anything on why you’re unable to create a staging site by using the GoDaddy staging site feature, nor we know why the WP Staging plugin failed to be installed on your site. We recommend you please reach your web hosting provider for further assistance to have the staging site ready for you. Once you have the staging site ready, you can perform the conflict check easily.

Now that I have WP Staging installed, how do I go about checking to see which plugins create a conflict with the sitemap?