Display by Category, Sort by Custom Field

I have a review style site, and on my category pages, I’d like to sort by overall score, but I can’t seem to figure out how to do it.

I thought I had it when I put this query in:

<?php query_posts('meta_key=rp_overall&orderby=meta_value&order=DESC'); ?>

But when I added a new category, I noticed my entire listing was there. So that basically just pulled everything with an rp-overall value, and ordered it by the value.

Is there a way I can also separate it by category?

My goal is to:

1. Separate by category
2. Order by the meta value of rp_overall

Thank you!