How to customize the robot.txt file?

  • Terry Apodaca

    (@terryapodaca)


    1 month ago

    In the FAQ sections it mentions below, but it confuses me….

    Can I customize the robots.txt file using this plugin?

    Yes, you can customize the robots.txt file using the Google XML Sitemap Generator for Google plugin. This will allow you to control which pages and directories search engines can crawl and index on your site.

    I cannot find anywhere in the plugin admin sections where I can customize the robots.txt file…unlike other plugins there are designated locations. I might be missing something with this “feature” or not totally understanding what you mean by being able to customize it.

    I know the file is being generated because I can see this in it:

    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    Sitemap: https://mysite.com/sitemap.xml
    Sitemap: https://mysite.com/sitemap.html

    But I need to add a couple more “Disallow: ” statements. How do I do that?

    The page I need help with: [log in to see the link]

Viewing 1 replies (of 1 total)
  • Plugin Author Auctollo

    (@auctollo)

    We don’t support editing of an actual file. When none is present we dynamically add it in order to add a sitemap hint. If you create an actual file you can add disallow statements.

Viewing 1 replies (of 1 total)
  • You must be logged in to reply to this topic.