We’ve finally managed to find the cause. It seems when users are having JavaScript errors (from other plugins), the AJAX calls to dismiss the notices aren’t working.
I’ve disabled the rate notice in latest version and will have to code a solution which will not use AJAX. Please check your console with the inspector of your browser for any errors. This is a serious problem, because all other plugins will also not be able to execute JavaScript code!
Let me know if I can help you out with anything.