Robotos.txt

  • i just need to ask is this roboto,txt is fien for my wordpress?

    User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /author
Disallow: /comments
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads
Alloww: /category/*/*

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

User-agent: Googlebot-Mobile
Allow: /

User-agent: ia_archiver-web.archive.org
Disallow: /

# digg mirror
User-agent: duggmirror
Disallow: /

Sitemap: https://omgbaby.net/sitemap.xml

    and wanna ask what this code does?

    Disallow: /*?*
Disallow: /*?

    Please note: i made changes to my robots.txt and allowed Categories,i need to ask did i done it right way?

Viewing 3 replies - 1 through 3 (of 3 total)
  • Thread Starter amir

    (@sanwal)

    here i have another example. please advice which one is better.

    Sitemap: https://www.yoursite.com/sitemap.xml

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /wp-admin/
    Disallow: /wp-icludes/
    Disallow: /trackback/
    Disallow: /archives/
    Disallow: /author
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.php$
    Allow: /wp-content/uploads

    User-agent: All
    Allow: /

    # Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*

    # Google AdSense
    User-agent: Mediapartners-Google*
    Disallow:
    Allow: /*

    User-agent: Googlebot-Mobile
    Allow: /

    User-agent: ia_archiver-web.archive.org
    Disallow: /

    # digg mirror
    User-agent: duggmirror
    Disallow: /

    Moderator Ipstenu (Mika Epstein)

    (@ipstenu)

    ?????? Advisor and Activist

    Read this: https://perishablepress.com/wordpress-robots-rules/

    SEO

    (@seo-india-professional)

    Disallow: /*.js$ instructs google and other search engines not to crawl any page with extension .js.

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Robotos.txt’ is closed to new replies.