Robots.txt file is not updating correctly in multisite

It all depends on whether you set the multisite up as sub domain or sub directory. If it’s sub directory then each site will have it’s own directory and you can place the robots.txt file in that directory.

AH I see, ok we had to add a custom robots.txt file to stop bots from hitting the site as much (using crawl-delay) and that is being readin instead of the virtual robots.txt file

Is there anyway you could add that option to the plugin so we could use site specific robot.txt files?

Do you mean adding crawl-delay as an option in our Robots.txt Generator module?

Yes indeed, that would be great. I’ve had sites taken down by bots and setting the crawl delay helped alot.