Website Rebuild with new theme and Gutenberg

If you do not materially change content or URLs, what is used to serve the content (theme, page builder, etc.) will not affect SEO rankings. Changing page builders and/or themes may or may not change page speed score. It depends on the specifics. I don’t think it would make significant difference in most cases.

While page rank is important, so is presenting attractive, meaningful content to end users. A page loaded with clever animations and graphics is not necessarily attractive and meaningful. A clean, lightweight theme can go a long way towards such a goal. As long as the theme isn’t bogging down your site, for theme selection I’d focus on appealing to end users over SEO. There are several plugins that can help you in implementing SEO measures.

You will not reasonably be able to make all the corrections Lighthouse recommends. Focus on the recommendations that will yield the biggest improvements or that are relatively easy to accomplish. Consider applying some of the measures presented Here:

Optimization

I believe the discourage search engines setting places a noindex meta tag on pages. That’ll be enough for the big search providers. There’s no telling what other providers will honor or not. No harm in doing both the discourage setting and robots.txt restrictions. Just don’t forget to undo them on launch ??

Hello @bcworkz,

And thank you for you response. I am in the middle of a big(?) problem right now. Unfortunately it seems I already got indexed after a few days (checked
site:https://www.psychologos-giannopoulou.gr/staging/ ).

I contacted bluehost cause they made my staging and I had told them 1000 times that my only concern is not to get found on google. The answer of an agent (I have talked with about a dozen so far) is that they had made a mistake with the robots.txt and hadnt included some denial code. He told me he added that yesterday but I dont know what to believe and where I stand right now.

I also checked at this:
https://support.google.com/webmasters/answer/6062608?hl=en&ref_topic=6061961&visit_id=637313942883646665-3424507720&rd=1
which got me a little more confused.

Is robots.txt a good method to block indexing or I should just have the “discourage etc” button checked on wordpress?

What am I supposed to do know? Run like a fool and delete staging website before things get worse? (they assured me that from yesterday it wont get indexed any more but I dont believe them). And of course forget about my rebuilding of the site?

Or believe them, keep it, finish my rebuilding at the staging site as soon as possible (within a few days), deploy and then delete the staging one and for all??

Do I need to make further actions with google? Or just deleting will let it go…I dont want to add it to my properties on search console and things get more complicated.

??

I only wanted something simple and now I feel like I am doomed because of the mistake of bluehost and dont know what to do

Are things bad? I dont know

• This reply was modified 4 years, 4 months ago by constangiann.
• This reply was modified 4 years, 4 months ago by constangiann.

Yeah, that article is confusing. If you “noindex” site pages as recommended, I don’t see any harm in robots.txt blocking as well, but apparently robots.txt blocking is not adequate by itself. Right now your site has meta tags to “index, follow”, where you want “noindex, nofollow”. The meta tags to index, follow are coming from Yoast SEO plugin. Unless the plugin has settings to use noindex, nofollow, you’ll need to deactivate the plugin and ensure the site has its “discourage indexing” option selected.

With the site showing noindex, nofollow, it wouldn’t matter what robots.txt does, at least as far as Google is concerned. But for the record, your robots.txt is currently not blocking the staging site. If you really want to block staging, you can edit the file yourself through your hosting account’s file manager. This is what the file’s content should be:

User-agent: *
Disallow: /staging/
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

I don’t have any recommendations regarding undesired search results. They’ll eventually become stale and not show up any more, but it could take quite a while. I think there’s a way to request removal of links through Google Search Console. Can’t say I ever had the need.

Thank you so much for your help,

I think you saw that because I unchecked the “discourage” setting for a few days thinking it was maybe that preventing the access to robots.txt. I checked it again.

Could you see again if it is now noindex?

As for Yoast SEO. When I check the settings->reading button Yoast puts a wide red notification at the top of wordpress saying:

“Important SEO Issue: Blocking access to the robots file. If you want search engines to show this site in their results, you must go to your Reading Settings and uncheck the box for Search Engine Visibility. I don’t want this site to show in the search results.”

So I assume I shouldnt do anything else…It seems global wordpress reading settings override Yoasts settings.

I am quite confused but I decided to finish my work quickly deploy and delete the staging. I don’t know what else to do.

As fot the robots.txt are you 100% sure? I am afraid to put some code there by myself. Not editing the file. That is easy for me. But putting code against what they put so I take the responsibility if it isnt right and the site access isnt blocked.If you are right then bluehost agents are dangerous.
??

They told me many times that the robots.txt they created is ok and I dont need to check the reading wordpress button and that I wont get indexed for sure.

• This reply was modified 4 years, 4 months ago by constangiann.
• This reply was modified 4 years, 4 months ago by constangiann.
• This reply was modified 4 years, 4 months ago by constangiann.
• This reply was modified 4 years, 4 months ago by constangiann.
• This reply was modified 4 years, 4 months ago by constangiann.

Within a portion attributed to Yoast, the site has “noindex, follow”, meaning the current page will not be indexed but Google can follow links on the page looking for other content to index. It’ll keep hitting noindex on other pages, so allowing follow is not necessarily a bad thing.
See https://www.psychologos-giannopoulou.gr/robots.txt
Nothing about disallowing /staging/. And https://www.psychologos-giannopoulou.gr/staging/robots.txt goes 404, which is correct, since the file belongs in the domain’s root public_html folder. I don’t know where Bluehost thinks they blocked /staging/ but it’s not where Google expects to find the file. Google will use the one first linked above.

You can attempt to get Bluehost to properly alter that file, or alter it yourself. The worst that can happen with a bad edit is search bots will ignore the file. There are lots of sites with no robots.txt, you’ll be no worse off than them. You are wise to avoid editing code files like wp-login.php because it can break the site. Errors in robots.txt only affect search bots, it won’t break anything, the site continues to work for regular users, and it’ll continue to be indexed if there is no “noindex” meta tag.

To reiterate, with “noindex” in place, you don’t really need to also block with robots.txt. “noindex” alone will stop Google from indexing.

Hello @bcworkz,

And thanks for everything. I didn’t reply earlier because I was finishing my staging. Yesterday bluehost deployed my staging to live and deleted my staging. It seems to be running and now my only concern is Google.

It seems that even now that the staging is gone when i put something of my staging sub urls in a browser like:

https://www.psychologos-giannopoulou.gr/staging/biografiko/

then it opens the live url:

https://www.psychologos-giannopoulou.gr/biografiko/

meaning there is some kind of redirect left somewhere. They are investigating and I am waiting for their email.

As for bluehost it seems that yesterday when I talked to them there was no robots.txt at all! Someone had deleted that without letting me know! This means: 1)someone from them created a robots.txt with my staging site 2)i got indexed 3)someone else told me the first robots was wrong and added some denial code and 4)someone else deleted the robots.txt without letting me know.

Not very good for customer care ??

Could you check if my running website is running and non blocking search engines right now?

My only concern is that and the redirects thing. I hope Google doesnt punish me for something which is not my fault.

What do you think?

Thanx
Constantenia

Your main site looks fine and is accessible to search bots AFAICT. If you really want to check for sure regarding Google, use the “Fetch as Google” tool within Google Search Console. If you’ve not used Search Console yet, I recommend you start. It’s important for anyone interested in their site’s search standing.
https://search.google.com/search-console/about

robots.txt is in place and only blocks /wp-admin/ (except for admin-ajax.php), which is appropriate for most sites.

The /staging/ part of the URL gets rewritten to the main site because WP ignores parts of URLs that don’t mean anything to it. Since the staging site no longer exists, the main WP installation takes control of the request. It finds a match for “biografiko” page slug and “/staging/” means nothing to WP, so it just serves up the matching page regardless.

You will not be penalized by this sort of behavior because both requests include identical canonical URLs in the head section. The staging URL sends a 301 permanently moved response, which is exactly what you want. This is much better than the link going 404 or something.

If you are right then I think my problems are solved and you helped me a lot with this.

I want to thank you so much for your interest and advice. It was like having a second eye with me. I was really anxious about the rebuild (still am) since my live site is ranking very good for years.

Google Pagespeed Insights brought me to this situation but I think it was for good. My score was 10-15% for mobile (with all the image optimization and caching) and now is 60% without having done anything yet (image optimization, caching etc). I think it will be much faster when I finish.

And it will be much easier to do refreshes of the website when I need to. I have seen now how Gutenberg works (I was afraid of it and still had Classic editor with WP Bakery) and I believe it is a better option.

Thanks for all
Constantenia

You’re welcome. That’s quite a speed improvement without taking further measures, nice! We’re glad you like the block editor. It’s always difficult to leave the familiar for the unknown, but that’s the path to improvement ??