Custom infinite scroll and archives/search

I have a file (loop.php) which contains the loop (while (have_posts … endwhile)
Next I have an index.php, archive.php and a search.php. All of which contain get_template_part(‘loop’).

In my functions.php I add a hook called wp_ajax_infinitescroll which executes the following function:

function MyInfiniteScroll(){
   $loop        = $_POST['loop'];
   $page           = $_POST['page'];
   query_posts(array('paged' => $page ));
   get_template_part($loop);
   exit();
}

Then I have some JavaScript which does (I left out some code here to simplify):

jQuery.ajax({
    url: "https://xxxxxxxxx/wp-admin/admin-ajax.php",
    type:'POST',
    data: "action=infinitescroll&page="+ pagenr + '&loop=loop',
    success: function(html){
        jQuery("#gids").append(html);
    }
});

My problem: on index.php this works perfectly, it shows all posts.
On both archive.php and search.php the first “page” works also, but any subsequent “pages” load all the posts again and not those limited to the search or archive.
I suppose somewhere along the line WP_Query is “reset”? So what am I doing wrong here/am I forgetting?

PS: before someone tells me there is an infinite-scroll plugin, I know. However it is not usable in this particular case which is why I try to write my own.