I know you said you tried it, but do you have it set to “Name Only” right now for the “Product Search” option? If it isn’t searching through the custom fields, then there really isn’t any reason the querying should take a long time, unless your server is just slow at handling database queries.
Also, looking at the source code, I can see that, even though they’re hidden, both the List and Thumbnail views are being loaded on the page. This means it is still running all the queries for those views (three times the amount of queries it needs to be running). How have you hidden them? The correct way is to do it in the shortcode, like so:
[product-catalogue id="X" starting_layout="Detail" excluded_layouts="Thumbnail,List"]