Robots.txt not getting updated

@sagarchaudhuri We’re sorry to hear you are experiencing this issue on your site.

If you clear your cache (server, plugin/theme, or CDN such as Cloudflare) does this resolve the issue? If you are not sure how to clear your server cache you should speak with your host provider.

We are using WP Plugin abd CloudFront CDN – we have deleted the cache at both places and checked. Still the robots.txt is not getting updated.

@sagarchaudhur` Can you please try?clearing your browser cache, cookies, and restarting your browser, check on different browsers? The most likely cause is cache from a server and sometimes browser cache.

@suascat_wp we have tried all those methods. Google Robots.txt tester is also reading the older version of the robots.txt

@sagarchaudhuri Do you have any other plugins that may be attempting to manage the robots.txt file? Also, are you able to check the file permissions of the robots.txt file using a file manager utility or FTP to confirm that the file has the correct read/write permissions?

@devnihil The only other major plugins that we are using are WP Rocket and Themify Builder.

@sagarchaudhuri Thanks for your reply.

Were you able to also check on the robots.txt file’s permissions via FTP?

@devnihil yes, I checked that. The permissions are fine. In fact, I deleted the robots.txt from server, still the same old file was visible when I access it through the browser or the Google tool. I created a new file using Yoast but still no change. There is some default setting somewhere.

Hi,

If the robots.txt file is not writable or your WordPress install has disabled file editing, creating or editing robots.txt through the WordPress Dashboard may fail. In that case, you can edit at the server level. WordPress generates a virtual robots.txt file if the site root does not contain a physical file. To override the virtual file, please follow these steps to create a physical robots.txt file.

• Use your favorite text editor and create a text file.
• Save the empty file with the name robots.txt.
• Upload the file to your server. If you’re unsure where on your server to upload the file, please contact your webhost.

If WordPress was blocking access to the virtual file, you should be able to edit the physical file from our plugin. If not, you can always edit the robots.txt directly on your server using FTP or a server file manager. Please contact your webhost for assistance if you have trouble uploading or editing files on your server.

You can refer to this article for more information – https://yoast.com/help/how-to-edit-robots-txt-through-yoast-seo/

If that still doesn’t work, we might need to check things further on your website, but we are unable to do so over these public forums. You may want to consider a subscription of Yoast SEO Premium in order to receive updates and support.

I edited the robots.txt manually – it overrode the virtual file used by Yoast.