Sort post by custom date then post_date if no custom date exists

I have a feed of posts and some of the posts can be categorised as events. The user can add event info via custom fields and one of the fields is event_date when they create a post that is an event. Other posts are news, blog etc. these all use the post_date.

I’d like to order my feed of all posts by the custom field event_date and this isn’t a problem but when I apply this it means the posts without a specified event_date are now in the wrong order. So what I want to do is order by two fields first the event_date and then post_date, is this possible?

I think I did something way back with a filter, is that possible route for a solution?
Any ideas are welcome.