I just wanted to explain why I need to change the robots.txt.
I received the following email from Google Adsense:
We noticed that our AdSense crawler is having difficulty crawling some parts of your site on https://www.xn--8dbbahx1a6dh.co.il. Because of this, your AdSense ads are less targeted and are generating less revenue on average. Specifically, we detected 97 failed crawl requests over a four day period earlier this month, which are caused by an issue within your robots.txt file. This file is blocking certain sections of your site that have AdSense code already on them from the AdSense crawler.
View Crawler Errors
Fortunately, you can fix this issue today by editing your robots.txt file and giving our AdSense ad crawler the ability to view the pages of your site with AdSense ad code on them. To do this, find your robots.txt file, located at https://www.xn--8dbbahx1a6dh.co.il/robots.txt and add the following two lines at the very top:
User-agent: Mediapartners-Google
Disallow:
