Archive page split up according to custom fields?

I’m going around in circles trying to figure this out so thought I’d give up and post instead, to see if anyone knows how to do something like this.

Okay, I have a custom post type called Press with approximately 400 articles so far. You can see my archive here: https://www.rachael-stirling.com/press/ and you can also see how long it takes to load. ??

I’m using the following code to generate the archive page:

<?php
    $args = array (
        'post_status' => 'publish',
        'post_type' => 'press',
        'posts_per_page' => -1,
        'meta_key' => 'wpcf-publication-date',
        'orderby' => 'meta_value_num'
    );
    $get_press = new WP_Query ($args);
    $current_year = '';
    while ($get_press->have_posts()) :
        $get_press->the_post();
        global $post;
        $this_year = get_post_meta( $post->ID, 'wpcf-publication-date', true );
        $this_year = date('Y', $this_year);
        if (($this_year != $current_year)) :
            if ($current_year != '')
                echo '</ul>' . "\n";
                echo '<h2 class="header">' . $this_year . '</h2>' . "\n";
                echo '<ul>' . "\n";
                $current_year = $this_year;
            endif;
            ?>
            <li>
                <span class="source"><?php echo(types_render_field("source", array())); ?></span> &mdash; <a href="<?php the_permalink() ?>" title="<?php the_title(); ?>"><?php the_title(); ?></a> <span class="publication-date">(<?php echo(types_render_field("publication-date", array("format" => "jS F"))); ?>)</span>
            </li>
    <?php
    endwhile;
    if ($current_year != '') echo '</ul>' . "\n";
    wp_reset_postdata();
    ?>

I’ve noticed when using SEO testers that the archive is timing out, probably due to the sheer volume of posts being returned.

Thing is, as you can see from my query, I’m using a custom field to order my posts (publication date) and splitting it up into years. I just don’t know how to do it differently.

I was thinking of maybe displaying a header for the current or most recent year, or the past 3 years or something, then having links below for the previous years.

E.g.

2013
Title 1
Title 2
Title 3

2012 | 2011 | 2010 | 2009 | 2008 | 2007 | 2006 <- links

Does anyone know how I would even start to do this, considering I’m splitting the sections up by custom field rather than post date? I know some internal functions exist for displaying archives by post date in this way but not sure about custom fields… hmmm…

Any guidance would be much appreciated. ?? Thanks.