robots.txt shows 404

  • Resolved umchal

    (@umchal)


    5 years, 3 months ago

    Hi there,

    I’m wondering if the plugin is configured properly. When I open the page https://{site url}/robots.txt, the 404 not found page appears. I searched the web for similar issues and it seems I’m supposed to see some rules in that page.

    The actual robots.txt file is not placed. I think it’s okay as the output is handled virtually.

    I deactivated all the other plugins and switched to the default theme Twenty Twenty. So no conflict is expected. I also resaved the Permalink settings (Settings -> Permalinks). Nothing helped.

    Any ideas?

Viewing 3 replies - 1 through 3 (of 3 total)
  • Thread Starter umchal

    (@umchal)

    Never mind. The site is installed in a sub-directory. In that case, the robot.txt cannot be dynamically displayed.

    
		// robots.txt -only if installed at the root
		$home_path      = parse_url( home_url() );
		$robots_rewrite = ( empty( $home_path['path'] ) || '/' == $home_path['path'] ) ? array( 'robots\.txt$' => $this->index . '?robots=1' ) : array();

    Maybe it would be nice to show a warning message to the user for such a case.

    Plugin Author Sybre Waaijer

    (@cybr)

    Hello!

    I’m glad you’ve found the issue!

    It’s true; the robots.txt standard only permits the file on the root directory. WordPress isn’t prone to notify you of issues–and I tend not to interrupt you from your work with TSF.

    Nevertheless, we did add an extra check and warning for that, in the robots.txt settings of the plugin:
    https://github.com/sybrew/the-seo-framework/blob/4.0.4/inc/views/admin/metaboxes/sitemaps-metabox.php#L145-L148

    It’s not all bad, because the robots.txt file doesn’t add much value to WordPress–this is because both TSF and WordPress halt robots via other means on pages they aren’t meant to crawl.

    In any case, I hope this helps! Have a nice day ??

    Thread Starter umchal

    (@umchal)

    That’s nice to hear. You too have a nice day!

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘robots.txt shows 404’ is closed to new replies.