Recent posts of same category on single post pages

I use this code to display 5 of recent posts in a single post view.

<?php
if (is_single( )) {
	$post_ID = $wp_query->posts[0]->ID;
	$all_cats_of_post = get_the_category($post_ID);
	for($i = 0; $i < sizeof($all_cats_of_post); $i++) { ?>
    <div class="recent-cat">
			<h3>Recently in <?php echo $all_cats_of_post[$i]->cat_name; ?>:</h3>
				<ul>
					<?php global $post; $cat_posts = get_posts('numberposts=5&category='.$all_cats_of_post[$i]->cat_ID); foreach($cat_posts as $post) : ?>

                    <li><a href="<?php the_permalink(); ?>"><?php the_title(); ?></a></li>

                    <?php endforeach; ?>
                </ul>
     </div>  

<?php } ?>

My question is: if I have a post under 2 categories or category/subcategory, how can I exclude a category or a subcategory?