A robot. txt file lives at the root of your site. So, for the site https://www.example.com, the robots. txt file lives at https://www.example.com/robots.txt.
Crawlers will always look for your robots.txt file at the root of your website, so for example https://www.contentkingapp.com/robots.txt. Navigate to your domain, and just add “/robots.txt”.
If nothing comes up, you don’t have a robots.txt file yet. Don’t worry, we explain here how to create a robots.txt file.
If you're working on a WordPress website that hasn't gone live yet, and you find the following contents in your robots.txt:
User-agent: *
Disallow: /
then check your settings under: Settings > Reading and look for Search Engine Visibility.
If the box Discourage search engines from indexing this site is checked, WordPress will generate a virtual robots.txt preventing search engines from accessing the site.
