file Robots.txt

  • Resolved fl0645

    (@fl0645)


    1 year, 8 months ago

    Hello, i have make a robots.txt but i don’t know if it’s good like that ?

    Thank you very much to your help,

    User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Allow: https://example.fr/
Allow: https://example.fr/mise-en-images-sons-realisation-audiovisuelle/
Allow: https://example.fr/services-audiovisuels/
Allow: https://example.fr/particuliers/
Allow: https://example.fr/contact/
Disallow : https://example.fr/*
    • This topic was modified 1 year, 8 months ago by James Huff. Reason: fixed formatting
    • This topic was modified 1 year, 8 months ago by Jan Dembowski.

    The page I need help with: [log in to see the link]

Viewing 4 replies - 1 through 4 (of 4 total)

  • Hi @fl0645

    First of all, this question isn’t really related to WordPress, but still.

    I had a look at the description from Google in https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt

    For Allow/Disallow statements, the URL should NOT include the protocol and domain name, but start with /.
    Within each applicable group (here you’ve got only one group, namely * = all bots).
    Rules are read from top and down until you get a match.
    So you specifically Disallow /wp-admin/ and anything under it.
    But then in the third line (after you fix it), you Allow “everything else”, so the bot will stop reading after this line…)

    Also: By default, everything is treated as “Allow” unless specifically “Disallow” has been applied. So no need to list several “allowed” locations.

    Perhaps you can describe in words what you’re trying to achieve?

    Thread Starter fl0645

    (@fl0645)

    Thank you very much for your answer, I try to optimize the site because it has a very low score in the search engines. I thought that by telling the mover to retrieve such page this would be positive for SEO because there are many “links” that I do not want them to appear as for example anything found in wp-content. I wrote a new text, is it better and functional than the previous one? Thank you very much for your help.

    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php
    Allow: /plprod74.fr/
    Allow: /plprod74.fr/mise-en-images-sons-realisation-audiovisuelle/
    Allow: /plprod74.fr/services-audiovisuels/
    Allow: /plprod74.fr/particuliers/
    Allow: /plprod74.fr/contact/

    An “Allow:” entry doesn’t in any way instruct Google to visit that page.
    Positively linking to various pages is instead done by adding a link to a sitemap.
    So rather something like this:

    User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://plprod74.fr/sitemap_index.xml

    More important is to think through how you’re structuring the content of your site, what internal links you put between various pages and that you actively work with SEO-marking of your content.
    Use any good SEO plugin you feel comfortable with, and read carefully their hints and support.
    Make sure to use https://search.google.com/search-console/

    Thread Starter fl0645

    (@fl0645)

    Hello, thank you very much, i try this now , have a nice day !

    • This reply was modified 1 year, 8 months ago by fl0645.
Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘file Robots.txt’ is closed to new replies.