Business Listings Not Indexed

Hi Jason Hsu!

Google does index MembershipWorks pages using Ajax crawling. For example, if you enter the following into the Google search box:

site:https://www.gwboardoftrade.com/ greens Kelowna

you will see that the description indicates:

“Search Clear. Search by location. Search by category. 19 Greens Kelowna Corp. 19 Okanagan Grill + Bar. 1st Class Auto Glass & Upholstery Ltd. 3BP Solutions.”

All the information in the description is generated by MW, so it is only possible if Google executes the Ajax and reads the HTML.

The issue may be your website host. Some hosting providers will alter the page behavior with robots. For example, WP Engine will redirect all URLs to the parent page (ie. the Business Directory page) and you have to disable that behavior manually:

https://wpengine.com/support/redirecting-bots-how-this-benefits-you/

If you are not using WP Engine, please check with your hosting provider as well as any plugins that may be causing this issue. I noticed that your web page does not load if I change my user agent to googlebot, so there is something unusual occurring there.

When you Google: site:https://www.gwboardoftrade.com/ greens Kelowna

The page that gets indexed is /business-directory/, and it gets indexed because it’s in the sitemap.

There is only one search result because the individual business listings aren’t getting indexed.

The listing for 19 Greens Kelowna Corp, is here:
https://www.gwboardoftrade.com/business-directory/#!biz/id/5baeafcdafd691881f962d30

This “page” generated using AJAX isn’t getting indexed, which is why if you Google searched for: site:https://www.gwboardoftrade.com/ “19greenskelowna.com”

It returned nothing.

I inspected “https://www.gwboardoftrade.com/business-directory/#!biz/id/5baeafcdafd691881f962d30” with Google Search Console, and it says the “URL is available to Google.”

With regard to the robot redirect issue you mentioned, even common browsers see links like /business-directory/#!biz/id/5baeafcdafd691881f962d30 as content on the same page, so I doubt they would get redirected, especially since it’s not even an integer that can be incremented at the end of the URL.

I still believe it’s an AJAX issue. Google crawls /business-directory/, but doesn’t follow the links which are just anchors. Without following the anchors, the info doesn’t get retrieved from MW via AJAX.

We need to get this resolved because having backlinks for SEO is one of the benefits of becoming a paying member. Please provide a solution.

Thanks

Hi Jason Hsu!

Perhaps a different example with one of our customers would be clearer. If you type “site:wtagc.org biz” into Google search box, you will see many indexed pages with the “Ajax” URLs (eg. https://wtagc.org/directory/#!biz/id/51ba5ef9316de6df46909ef9) with the member’s name, city/state and short description.

So Google is capable of indexing our Ajax pages, and our system has been designed from the start to support Google’s Ajax crawling. Please refer to these articles as confirmation:

https://developers.google.com/search/docs/ajax-crawling/docs/learn-more
https://webmasters.googleblog.com/2017/12/rendering-ajax-crawling-pages.html
https://searchengineland.com/google-can-crawl-ajax-just-fine-322254

In this case something is blocking Google from indexing your directory profile pages. If you are using WP Engine please see the linked article on how to disable the robot redirect behavior.

https://wpengine.com/support/redirecting-bots-how-this-benefits-you/

This site is hosted on SiteGround, not WPEngine, but I created a ticket to check with the hosting provider. I suspect they’ll probably say the problem isn’t on their end.

Instead of having two vendors pointing fingers at each other, can you provide a way to create a sitemap for the business listings? We can manually submit it to Google Search Console to make sure Google crawls through them all.

Thanks

Hi Jason Hsu!

There is no way to automatically generate a sitemap but a sitemap would not fix the issue here – as you mentioned that you tried to fetch a URL manually using Google Search Console it said the “URL is available to Google.”. So even if Google is provided with all the URLs (through a sitemap), it is unable to index those pages.

Sorry but I have to disagree.

I googled: inurl:”gwboardoftrade.com/business-directory”, and three of the listings were indexed. They are from the 15th, 23rd and 36th page of the directory, so quite old, but they were indexed at some point.

When I inspected this page that isn’t indexed in Search Console: https://www.gwboardoftrade.com/business-directory/#!biz/id/5df4af30f033bff57038bdc0

This is what was reported:

URL is not on Google
This page is not in the index, but not because of an error. See the details below to learn why it wasn’t indexed.

Sitemaps: N/A
Referring page: None detected
URL might be known from other sources that are currently not reported
Last crawl: N/A
Crawled as: N/A
Crawl allowed?: N/A
Page fetch: N/A
Indexing allowed?: N/A
User-declared canonical: N/A
Google-selected canonical: N/A

When I inspected this page, that IS indexed: https://www.gwboardoftrade.com/business-directory/page/36/#!biz/id/5d3914b5afd691ba2f8ec68c

URL is on Google
It can appear in Google Search results (if not subject to a manual action or removal request) with all relevant enhancements.

Sitemaps: N/A
Referring page: None detected
URL might be known from other sources that are currently not reported
Last crawl: Jan 28, 2020, 3:27:57 AM
Crawled as: Googlebot smartphone
Crawl allowed?: Yes
Page fetch: Successful
Indexing allowed?: Yes
User-declared canonical: None
Google-selected canonical: Inspected URL

In both cases, there are no referring pages, which suggest that Google doesn’t see the links on the main page as referring links.

So based on that, I’d like to believe that if there was a sitemap, they would get indexed.

I also don’t believe that there’s no way to automatically generate a sitemap. Perhaps you meant the plugin isn’t currently programmed to generate a sitemap, which I understand. However, from a developer to another, we both know that there’s a way to generate a sitemap if MembershipWorks wanted to.

Please, help your client out here and provide a solution.

Thanks

Reply from SiteGround support:

Hello,

Thank you for contacting our Technical Support!

I reviewed your case and as I can see there are no limits on our end which would block the GoogleBot from crawling a specific page.

I checked the access logs and as I can see there are no logs of the GoogleBot trying to crawl a business-directory/page:

I checked if there are any logs for both trying to access pages such as /business-directory/page/36/, yet there are none at all. There are no logs for the bots even trying to crawl such pages.

With this being said I would conclude that the issue is not caused due to a limit on our end. Indeed, the issue may be caused due to an AJAX call or conflict between plugins.

This is why I would advise you to stop all plugins except the MembershipWorks plugin one by one and check after each if the issue is still present. This way you will find the faulty plugin which you can then replace with another one having the same functionality or you can reach out the plugin’s developers to review the matter on their end.

If you need any further assistance don’t hesitate to contact us again.

Best Regards,

Bozhana Tringova
Senior Technical Support

So, Google was able to index the /business-directory/ page, but didn’t recognize it as a referring page for all the individual business listing pages.

Hi Jason Hsu!

To help with diagnosis we’d need some tools/access:

1. A WordPress admin login so we can check/disable other plugins as necessary, as well as fix some HTML structure issues on your website (which can cause indexing issues).

2. Add us as a user to the Google Search console for https://www.gwboardoftrade.com so we can fetch the links as Google. We cannot use the “fetch as Google” feature without being a user on the domain’s search console.

You can remove the WP admin and Google Search Console users as soon as we’ve completed the diagnosis. Please send the access credentials to us at [email protected].

Thanks for taking a closer look at this. I sent an email to [email protected].