The thing is, each server the emails passes through may hold on to it for a while. Most are quick, a couple of minutes typically. Some rare ones only process mail once per night. Once it’s left your server, you have no control over how long it takes. The ISP receiving the email has a hand in this.
You said that email to other server users are going through quickly. Perhaps email being sent out of the server are being queue’d, spam/virus scanned etc. Or perhaps the receiving ISP’s don’t trust the source, because your server lacks a reverse dns record. There are many reasons an email may get put through a queue for extra spam scanning. These queue’s usually process in batches, once every so often. That’s where delays typically come from.
Or heck, there might even be a spotty dns problem. The mail server can’t connect, and puts it aside to try later. I could use up all my fingers and toes and still not relate all the possible reasons email might get delayed.
