Problems with post__not_in function and archive pages

I have WP site with 3 articles from one category (ID 4) printed in big, nice rotator and rest is showing in normal list. I want to skip these 3 last published posts from cat ID 4 in normal list.

I put together piece of code to do it

<?php
query_posts('cat=4&posts_per_page=3');
$ids = array();
while (have_posts()) : the_post();
$ids[] = $post->ID;
endwhile;
?>		

<?php
query_posts(array('post__not_in' => $ids));
while (have_posts()) : the_post();?>
<li>
<?php the_title();?>
</li>
<?php endwhile;
?>

In first loop I fill array $ids with 3 ID of articles and in second I list all skipping these three from array. It works correctly on first page (homepage), but as soon as I go to archive and try to list older posts (page 2+), it shows same content as on first page. Interesting fact is, that it correctly count amount of articles and generates correctly number of pages with history. Only problem is, that all of them showing same content as first page.

Any idea what to improve?