I am having the same issue. It slows down waiting time of user listing of 20 people by ~6 seconds. Once you untick gravatar support, the page loads in 0.4s.
I am not sure why you need to do the requests on the server side, as it makes more sense to check if there is a local avatar; and if there is none load gravatar on the client side.
