SEO Title and Meta Description not showing in Search Engine

@webmastert We’re sorry to hear you are experiencing this issue on your site.

We checked on your site’s homepage and do see the ‘No information is available for this page’ message for it in Google’s search results.

We verified the page source of your site homepage has no ‘noindex’ tag as expected. We went ahead and checked the robots.txt file. Per Google?here, robots.txt file might be the possible cause.

We do recommend you edit the file and just have this standard rule for better crawlability of the site.

User-Agent: *
Disallow: /wp_admin/
Allow: /wp-admin/admin-ajax.php

This would allow GoogleBot to crawl all the links on your site and better understand how the pages are viewed and displayed. This guide explains more: https://yoast.com/ultimate-guide-robots-txt/. We have additional recommendations on robots.txt here: https://yoast.com/wordpress-robots-txt-example/.

Once done, we recommend that you submit the Yoast-generated sitemap https://film-book.com/sitemap_index.xml and to delete any other sitemaps. In addition, please follow these steps:

• Please submit your website to the Google search console, if you haven’t done so, by following the steps given in <u>this guide.</u>
• Please submit your sitemap to your Google search console, if you haven’t done so, by following the steps given in <u>this guide</u>.
• You can then try to use “Fetch as Googlebot” on your homepage and verify if the search results for your site are updated.

It might take some days for Google to fully crawl your site and index URLs in the search result.

Hi Support,

1.) I went through the steps as you suggested, altering the robots.txt file, etc.

I added the sitemap to Google Search Console and this error showed up:

https://capture.dropbox.com/gHuOO2uGc1SPph8x

"Sitemap could not be read
General HTTP error
1 instance
We encountered an error while trying to access your sitemap. Please ensure that your sitemap is present at the specified address and is not blocked to Google. See our help center for more debugging help."

What did I do wrong?

2.) Also, I ran a search of film-book.com in the URL Inspection Tool and found this:

https://capture.dropbox.com/4G6l9mUhaCxymljH

"Indexed, though blocked by robots.txt
Crawl allowed? No: blocked by robots.txt
Page fetch: Failed: Blocked by robots.txt"

I don't want Google crawling or page fetching blocked. I was told by my web host that we had slowed down the google bot crawl rate, not stopped it all together. Google seems to be saying otherwise. Am I wrong?

3.) I clicked Live Test in Google Search Console, and received this result:

https://capture.dropbox.com/sCqSlXU9GzYpWQTm

"URL is not available to Google
This page cannot be indexed. Pages that aren't indexed can't be served on Google. See the details below to learn why it can't be indexed Learn more"

"error
Page availability
Page cannot be indexed: Blocked by robots.txt
URL will be indexed only if certain conditions are met
Crawl
Time
Jan 29, 2023, 9:54:04 AM
Crawled as
Googlebot smartphone
Crawl allowed?
error
No: blocked by robots.txt
Page fetch
error
Failed: Blocked by robots.txt
Indexing allowed?
N/A"

You said that "It might take some days for Google to fully crawl your site and index URLs in the search result." but if Google's bots can't crawl the pages, how are these errors going to change?

4.) I have had Yoast SEO installed for years now. Do I need to go through the plugin's "SEO data optimization" under "First-time configuration"?

Thank you.

Hello @webmastert, I’ve addressed your concerns below:

1. Sitemap could not be read – General HTTP error.
   This could be a temporary error as the sitemap index loads fine in a web browser, and no issues are detected in the sitemap validator. Please check whether the sitemap can be fetched using the URL Inspection Tool. Here’s a guide
2. Indexed, though blocked by robots.txt
   Do you mean you do not want Google crawling your homepage?
3. Crawl allowed? No: blocked by robots.txt
   The screenshot shows that https://film-book.com is not on Google; it has not been added to their index. Is this not what you want?
4. SEO data optimization
   Yes, it would be best if you ran the analysis via the First-time configuration, or you can also access it in WordPress > Yoast SEO > Tools

We look forward to hearing from you.

Hi Maybellyne,

Thanks for the quick reply.

1. ) I did as you suggested. I placed the sitemap (https://film-book.com/sitemap_index.xml) in the URL Inspection Tool and this was the result:

URL is not on Google
This page is not indexed. Pages that aren’t indexed can’t be served on Google. See the details below to learn why it wasn’t indexed.

Page indexing
Page is not indexed: Excluded by ‘noindex’ tag

Discovery
Sitemaps
N/A
Referring page
https://film-book.com/tag/jason-tobin/
https://film-book.com/upload-amazon-studios-renews-the-sci-fi-comedy-tv-series-for-a-second-season/
https://film-book.com/tv-review-animal-kingdom-season-2-episode-9-custody-tnt/
https://film-book.com/ready-player-one-2018-movie-trailer-4-final-promo-steven-spielbergs-film/
Crawl
Last crawl
Dec 21, 2022, 5:21:02 AM
Crawled as
Googlebot smartphone
Crawl allowed?
Yes
Page fetch
Successful
Indexing allowed?
info
No: ‘noindex’ detected in ‘X-Robots-Tag’ http header
Indexing
User-declared canonical
None
Google-selected canonical
?
Inspected URL

I followed the guide that you linked to. I clicked “Request Indexing” for the sitemap (https://film-book.com/sitemap_index.xml). The result was:

Indexing request rejected
During live testing, indexing issues were detected with the URL

What do I do now?

2.) I do want Google crawling my home page, fetching my home page, and all of the other pages and posts on the site.

3.) I want FilmBook (https://film-book.com) added to Google. It was previously. How do we accomplish this…again? How did the site get un-added?

4.) Okay. Will do.

4. Update – I ran the SEO data optimization and received this error:

Error details

Error message
Error parsing the response to JSON.

What should I do?

• This reply was modified 1 year, 10 months ago by WebmasterT.

4. Update 2 – I tried running it again and this was listed under Response both times (attached). What do I do?

Hello @webmastert,

1. It means Googlebot could access the sitemap, though you shouldn’t request indexing for a sitemap. What happened previously was a temporary issue. So you should re-submit the sitemap.
2. If you want Google to crawl, fetch and index your entire site, your current robots.txt file is not in line with recommended best practices.
3. To ensure https://film-book.com/ is indexed by Google, please edit your robots.txt file, as you’re blocking the page with the directives in the file.
4. To troubleshoot further while you’re on the SEO Data Optimization screen, can you please check if there are Javascript errors on your browser’s console? You may follow the steps here:?https://yoast.com/help/how-to-find-javascript-errors-with-your-browsers-console/

1.) I resubmitted the sitemap and this was the result (https://prnt.sc/EOl1GCb4xO__):

Sitemap could not be read
General HTTP error
1 instance
We encountered an error while trying to access your sitemap. Please ensure that your sitemap is present at the specified address and is not blocked to Google. See our help center for more debugging help.

How do I submit a sitemap (https://film-book.com/sitemap_index.xml) that can be read by Google Search Console?

2-3.) This robots.txt has been implemented:

User-Agent: *
Disallow: /wp_admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://film-book.com/sitemap_index.xml

Does this allow Google to index the site?
Does this allow Google to crawl, fetch and index your entire site?

4.) I looked at the browser's console as you instructed. Here is the result (https://prnt.sc/rmiJh6PBBEh1):

JQMIGRATE: Migrate is installed, version 3.3.2
admin-bar-v2.js?ver=11.7.1-202305:3 Missing data from PHP (wpNotesArgs).
(anonymous) @ admin-bar-v2.js?ver=11.7.1-202305:3
tie.js?ver=1675185723:23 Page Template Attr: select[name='page_template']
?v=2.0:16 WebSocket connection to 'wss://public-api.wordpress.com/pinghub/wpcom/me/newest-note-data' failed:
v @ ?v=2.0:16

What do I do? What is the next step? 

• This reply was modified 1 year, 10 months ago by WebmasterT.

Hello @webmastert

Sitemap: General HTTP error
It’s unclear why Google Search Console reports that error while Googlebot could fetch it when you used the URL Inspection Tool. Again, the?sitemap index?loads fine in a web browser, and no issues are detected in the?sitemap validator. Please post about this in the Google Search Help Community to receive more insights.

Robots.txt
The file is following the recommended best practices and there’s nothing else for you to do on it.

SEO data optimization
We have seen unexpected issues where the previously optimized data is invalid or incomplete. So I recommend resetting the data, and our?How to reset the Yoast indexables?guide will walk you through the process step by step. We’ve also summarized the steps below:

1. Install & activate the Yoast Test Helper plugin
2. Go to Tools > Yoast Test
3. Locate the Yoast SEO section and click on the ‘Reset indexables tables & migrations‘, ‘Reset Prominent words calculation‘, and ‘Reset Internal link counter‘ buttons. After each click, the page will reload to confirm that the specific reset was successful.
4. Go to SEO > Tools, and under SEO data, click the “Start SEO data optimization” button to allow Yoast to rescan your content.

Do let us know how it goes.

SEO data optimization

I followed all four steps. The optimization got about half complete before the error occurred again (https://prnt.sc/VsQ6-9FLaFA-):

• Oops, something has gone wrong and we couldn’t complete the optimization of your SEO data. Please click the button again to re-start the process.Below are the technical details for the error. See?this page?for a more detailed explanation.Error detailsError message
  Error parsing the response to JSON.Response

What is the next step?

Hello @webmastert,

I am sorry about the error you received when running the SEO data optimization. The error above is generic, and I cannot describe what could have caused it. From time to time, we see that other site components could prevent SEO data optimization from running smoothly. Therefore as a first step, we ask you to try the following:

1. Install the Health Check plugin. The plugin will permit you to perform a conflict check by deactivating plugins and switching themes. Doing so will not affect the SEO of your site, and your visitors will still be able to view your site as usual.
2. Verify you have the default theme Twenty-Twenty. If not, please install it.
3. Navigate to WordPress > Tools > Yoast Test
4. Click on the ‘Reset indexables tables & migrations,’ ‘Reset Prominent words calculation, and ‘Reset Internal link counter’ buttons. After each click, the page will reload to confirm that each reset was successful.
5. Activate the Troubleshooting Mode (TM) in the Health Check plugin.
6. In TM, activate only the Yoast SEO plugin and the default theme Twenty-Twenty.
7. Rerun the SEO data optimization.
8. Deactivate the Troubleshooting Mode in the Health Check plugin.

In this way, was the SEO data optimization completed? Do let us know.

Hi Support,

I followed all of the steps and received this error:

Oops, something has gone wrong and we couldn’t complete the optimization of your SEO data. Please click the button again to re-start the process.

Below are the technical details for the error. See this page for a more detailed explanation.Error details

Request URL
https://film-book.com/wp-json/yoast/v1/indexing/prepare

Request method
POST

Status code
403

Error message
Cookie check failed

What do I do now? What is the next step?

The first time I ran it, this was the message (The Request URL) was different.

Thank you.

Thanks for following up @webmastert

The pathways may differ, but the status code is the same, error 403. When requesting information from the WordPress REST API, your server returns a 403 Forbidden. We received a similar report from other users where their web host’s security measures caused the issue.

Please ask your web host to update the ‘mod_security‘ settings that prevent our plugin from accessing the REST API. If there are no ‘mod_security’ settings preventing access at the server, you may have a custom .htaccess rule or security plugin that limits access to the WordPress REST API.

If you have a Web Application Firewall (WAF)s, like Comodo, please verify that the server does not remove the parameter from the Yoast REST API requests. Some users have reported that some WAFs setups are too strict in preventing Blind SQL injection attacks. Your web host or server admin can help you if you’re not familiar with WAF settings on your site.

Would any settings in your server/firewall explain this behavior? If you don’t know how to check this, please reach out to your server administrator or web host. They’ll be able to review your server setup for you.

Hi Maybellyne,

I gave your message to my web host and this was their reply:

I just wanted to give you a quick update on this ticket. After several hours, the SEO optimization finally failed on my end. Unfortunately, I got a slightly different error from yours. Mine was a generic 500 POST request generated when the optimization failed. I went ahead and attached some screenshots of the exact error from Yoast, which was simply "Error message Error parsing the response to JSON". However, using the Health Check plugin to disable iThemes may have resulted in me seeing something other than 403. But the 500 status I was getting was from https://film-book.com/wp-json/yoast/v1/link-indexing/terms. The exact timestamp from DevTools was "Sun, 05 Feb 2023 01:04:23 GMT".

I did cross reference the server-side PHP logs and the website transfer logs, both of these showed 0 relevant server-side log entries. Even searching the transfer logs for the exact timestamp only showed 2 successful 200 requests from this URL. Here are the DevTools request headers and the transfer logs:

Request URL: https://film-book.com/wp-json/yoast/v1/link-indexing/terms
Request Method: POST
Status Code: 500
Remote Address: 172.67.129.226:443
Referrer Policy: strict-origin-when-cross-origin
cf-ray: 7947bb53fc8c6312-ORD
date: Sun, 05 Feb 2023 01:04:23 GMT
server: cloudflare
vary: Accept-Encoding
:authority: film-book.com
:method: POST
:path: /wp-json/yoast/v1/link-indexing/terms
:scheme: https
accept: /
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
content-length: 0
cookie: wordpress_test_cookie=WP%20Cookie%20check; wordpress_logged_in_ba73c337561b427366fdf8d0ae57f3e0=nexcess_support_63dd2ba318389%7C1675717165%7CNMgib4CiPA6d9c9fAFK4NFIYVdAUWHmQrlig1sImcWI%7C982c1da2eae56c08fd26b89707d0e05eed08ff04226f78e3b455b537cb246d66; wp-health-check-disable-plugins=fec7df89c9d2a8a9133ae10559fa7be9; wp-settings-time-154=1675557534
origin: https://film-book.com
referer: https://film-book.com/wp-admin/admin.php?page=wpseo_tools
sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-origin
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36
x-wp-nonce: cc431a2572

[0128][[email protected] logs]$ grep '05/Feb/2023:01:04:23' transfer-2023-02-05.log | grep '50.28.76.132'
50.28.76.132 - - [05/Feb/2023:01:04:23 +0000] "POST /wp-json/yoast/v1/link-indexing/terms HTTP/1.1" 200 6872 "https://film-book.com/wp-admin/admin.php?page=wpseo_tools" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
50.28.76.132 - - [05/Feb/2023:01:04:23 +0000] "POST /wp-json/yoast/v1/link-indexing/terms HTTP/1.1" 200 6892 "https://film-book.com/wp-admin/admin.php?page=wpseo_tools" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"

Now, I do see that the site is also behind CloudFlare. So the 500 (and 403 from earlier) may be generated from something on the CDN side. But from what I can see the SEO optimization errors appear to be triggered by something either WAF related or application related. Additionally, I did notice that that your site currently has debug mode enabled. I did cross-reference that same timestamp in the debug logs and see a large number of WordPress database errors logged for your Easy Social Share Buttons plugin. I'm seeing database queries erroring out from the acd1ffcc_wrdpnew.wp_essb_subscribe_conversions table. So these sporadic errors could be related to database query issues. Here are the log entries around this timestamp:

[0123][[email protected] wp-content]$ grep '05-Feb-2023 01:04:2$*' debug.log
[05-Feb-2023 01:04:21 UTC] WordPress database error Table 'acd1ffcc_wrdpnew.wp_essb_subscribe_conversions' doesn't exist for query SHOW FULL COLUMNS FROM wp_essb_subscribe_conversions made by require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), do_action('template_redirect'), WP_Hook->do_action, WP_Hook->apply_filters, ESSB_Ajax::do_ajax, do_action('essb_ajax_subscribe_conversion_loaded'), WP_Hook->do_action, WP_Hook->apply_filters, ESSB_Subscribe_Conversions_Pro::db_log_loaded, ESSB_Subscribe_Conversions_Pro::db_log
[05-Feb-2023 01:04:24 UTC] WordPress database error Table 'acd1ffcc_wrdpnew.wp_essb_subscribe_conversions' doesn't exist for query SHOW FULL COLUMNS FROM wp_essb_subscribe_conversions made by require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), do_action('template_redirect'), WP_Hook->do_action, WP_Hook->apply_filters, ESSB_Ajax::do_ajax, do_action('essb_ajax_subscribe_conversion_loaded'), WP_Hook->do_action, WP_Hook->apply_filters, ESSB_Subscribe_Conversions_Pro::db_log_loaded, ESSB_Subscribe_Conversions_Pro::db_log
[05-Feb-2023 01:04:24 UTC] Failed to insert log entry: WordPress database error: Processing the value for the following field failed: url. The supplied value may be too long or contains invalid data.
[05-Feb-2023 01:04:24 UTC] Failed to insert log entry: WordPress database error: Processing the value for the following field failed: url. The supplied value may be too long or contains invalid data.
[05-Feb-2023 01:04:24 UTC] WordPress database error Table 'acd1ffcc_wrdpnew.wp_essb_subscribe_conversions' doesn't exist for query SHOW FULL COLUMNS FROM wp_essb_subscribe_conversions made by require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), do_action('template_redirect'), WP_Hook->do_action, WP_Hook->apply_filters, ESSB_Ajax::do_ajax, do_action('essb_ajax_subscribe_conversion_loaded'), WP_Hook->do_action, WP_Hook->apply_filters, ESSB_Subscribe_Conversions_Pro::db_log_loaded, ESSB_Subscribe_Conversions_Pro::db_log
[05-Feb-2023 01:04:24 UTC] Failed to insert log entry: WordPress database error: Processing the value for the following field failed: url. The supplied value may be too long or contains invalid data.
[05-Feb-2023 01:04:24 UTC] Failed to insert log entry: WordPress database error: Processing the value for the following field failed: url. The supplied value may be too long or contains invalid data.

What is the next step?

Thank you.