WordPress Taxonomy Template Pagination Issue

After dedicating hours on this issue I finally decided to ask the community about the problem I am facing. This issue itself is somewhat interesting and strange.

I created a custom taxonomy template and while I am using the default wordpress loop I am facing issues in Pagination. The posts in every page are coming randomly.

So lets say if I am on page 1 of the taxonomy pagination it shows me 2 posts and if I go to page 2 it shows me 4 posts then 2 and then 3 posts per page ..

I have set the posts_per_page as 6 using pre_get_posts action on my functions.php file ..

I am giving the brief codes below ..

TEMPLATE FILE

<?php get_header(); ?>
<?php
global $wp_query;
$big = 999999999; // need an unlikely integer
$paged = ( get_query_var('paged') ) ? get_query_var('paged') : 1;
?>

<div <?php post_class('main-section page-inner-wrapper'); ?>>
    <?php get_template_part('talent_filter', 'talent_filter.php'); ?>
    <?php if ( have_posts() ) : ?>

            <?php while ( have_posts() ) : the_post(); ?>                                       

                <!-- POST HTML -->

            <?php endwhile;  ?>

            <!-- PAGINATION BEGIN -->
            <nav class="navigation pagination" role="navigation">
                <?php
                    echo paginate_links( array(
                        'base' => str_replace( $big, '%#%', esc_url( get_pagenum_link( $big ) ) ),
                        'format' => '?paged=%#%',
                        'current' => $paged,
                        'prev_text' => __( '<i class="fa fa-angle-left"></i>', 'bridge' ),
                        'next_text' => __( '<i class="fa fa-angle-right"></i>', 'bridge' ),
                        'before_page_number' => '<span class="meta-nav">' . __( '', 'bridge' ) . ' </span>',
                        'total' => $wp_query->max_num_pages
                    ) );
                ?>
            </nav>
            <!-- PAGINATION END -->

      <?php endif; ?>
</div>
<?php get_footer(); ?>

FUNCTIONS IN function.php

function my_post_queries( $query ) {
  // do not alter the query on wp-admin pages and only alter it if it's the main query
    if (!is_admin() && $query->is_main_query()){

    // alter the query for the home and category pages 

        if( is_tax( 'talent_groups' ) ) {
            $query->set('posts_per_page', 6);
            $query->set('posts_per_archive_page', 6);
            $query->set('orderby', 'date');
        }
    }
}
add_action( 'pre_get_posts', 'my_post_queries');

// Solves Pagination Pages 404's
add_filter( 'option_posts_per_page', 'tdd_tax_filter_posts_per_page' );
function tdd_tax_filter_posts_per_page( $value ) {
return (is_tax('talent_groups')) ? 1 : $value;
}