Zoom defaults to wrong variation image

I have code that changes the product image programmatically on a variable product. I’m using the woocommerce filter ‘woocommerce_product_get_default_attributes’ to set the default attribute values and that’s working. The product image changes to the correct image for the variation. However, the zoom window shows a closeup of what would have been the product image if I did nothing. I’ve tried removing the product image and that didn’t work. Any help would be appreciated.