As shown in the documentations for ‘previous_post_link’ and ‘next_post_link’ both functions can take four parameters:
$format
$link
$in_same_cat
$excluded_categories
As you should be able to guess you want to set the third one, (which excepts a boolean of either true or false) to false, which will allow your to browse irrespective of category.
<?php previous_post_link( '%link', '<div class="meta-nav"> ← %title</div>', false ); ?>
previous_post_link and next_post_link