I don’t think it’s really “prioritizing” per se, rather it’s listing whichever data comes back first. Separate API search requests are made for posts, terms, formats, and media. If searching for posts takes a long time, media results could come back first and thus get listed first.
I’m completely speculating based on observation of behavior on my site. I don’t have a lot of posts, so that query doesn’t take so long. I see post results first. You should be able to see that posts are queried first by examining API requests logged in your browser’s network developer tool.
If this is indeed what’s happening, I’m afraid there’s not a good solution. The link search script would need to be overwritten to have it wait for post results before continuing on to the others. This would significantly increase the time needed to get full results.