[Robots.txt] What is the correct User agent for Adsense Crawl

  • Happen to face Google adsense crawl error, found the fix.

    It says, to make adsense to crawl your pages and show targetted ads, add these lines on top of robots.txt.

    User-agent: Mediapartners-Google
Disallow:

    But my robots.txt file has already had this lines

    User-agent: Mediapartners-Google*
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

    Someone explain me what these lines refer to and what is the difference between both.

    Which is the right one? Do I need to make any changes or just leave like that?

Viewing 4 replies - 1 through 4 (of 4 total)
  • Thread Starter Srivathsan G.K

    (@dangerdiabolick)

    Bump. Someone help ??

    Hi there,
    the easiest solution is to put:

    User-agent: *
    Disallow:

    at the top, and remove the rest. This will allow everything to crawl your site, and comes with that warning.

    You can read more about robots.txt here:
    https://en.wikipedia.org/wiki/Robots_exclusion_standard

    because the topic is too verbose to discuss here.

    Best of luck!

    Thread Starter Srivathsan G.K

    (@dangerdiabolick)

    Hey,

    thanks for the quick reply.

    I just want to ask only one thing.

    Do I need to replace the 4 lines of code or can I leave it as it is?

    There is no problem with indexing.

    I am just asking only for adsense crawlers.

    My robots.txt already has these 4 lines of code

    User-agent: Mediapartners-Google*
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

    Do I need to replace it with 

    User-agent: Mediapartners-Google
Disallow:

    Or just leave the default one as it is.

    Thanks.

    Just use code below as reccomended by google using string [*] will cause problem with some crawler as i experience before.
    Below is the latest standard used by adsense crawler.

    User-agent: Mediapartners-Google
    Disallow:

    Further more if you can delete all the instruction on txt below no need for it becos it interfere other agent access. No need for code below. Hope it help.
    User-agent: Googlebot-Image
    Allow: /wp-content/uploads/

    User-agent: Adsbot-Google
    Allow: /

    User-agent: Googlebot-Mobile
    Allow: /

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘[Robots.txt] What is the correct User agent for Adsense Crawl’ is closed to new replies.