You really need to re-think your user experience. I understand the desire to offer the ability to alter every aspect of a theme, that desire is why I learned to hack WP. Honestly, most users don’t care that much, they just want a nice looking site. In fact, letting most users alter every aspect of your theme will destroy the cohesiveness of your theme’s design. Offering more options is actually a disservice to most users. They are not graphic designers, letting them design their own site through many settings will not end well, even though they got exactly what they want. As non-designers, what users want is not good enough.
You don’t need to totally get rid of most of your settings, just move them out of the customizer to a traditional settings page. Consider the settings on the traditional page as rarely used “overrides”. If you are familiar with the Firefox browser, you know the most common settings are accessed through the Preferences dialogs, but users can also change hundreds of settings through the quasi hidden about:config
dialog. Your theme could do something similar.
For example, a customizer setting could manage the color of all headings through out the site, except specific headers that have overrides. Most users will be happy to leave the header overrides as “default”, meaning the customizer setting is used. Those so interested can still override a specific header color, all other headers continue to use the customizer settings.
With a traditional settings page, you can store all the settings in a single array, stored in the DB as a single option value. Instead of making dozens of queries to get the settings, you only do one. Granted, a lot more data needs to be transmitted, but no more than the individual settings added together. One query instead of dozens will save a huge amount of time! To be fair, the Customizer stores settings the same way, but these settings also involve a lot of overhead. By off loading the bulk of settings outside the Customizer, this overhead is reduced.
Another inefficiency is the customizer CSS is output in the head section to override any defaults from style.css. By moving little used settings outside the customizer, you can manage the CSS with a dynamic CSS page where the CSS file content is generated on the fly with PHP. The browser gets the proper CSS the first time without a bunch of overriding CSS in the head section.
This approach will not help refresh speeds in the customizer, in fact, it may be worse. You will also spend more time writing code to work the external settings into the customizer preview, not to mention writing code to manage the external settings themselves. However, the load speed of regular page requests will be much faster.