it seems that his happens whenever the server needs to query the downloads list, for example when you try to include a download into a post with the ?Insert Download? Button from the editor.
I should also mention, that I did not had this problem when I started using it, even if the “All Downloads” page was slow to load from the beginning. I have also bought the S3-Add-On and it seemed to me, that the problem started after using it, but disabling the add-on does not change anything.