@conciseac, no, you don’t have to unload all CSS/JS, but to check which files (belonging to specific plugins or within the active theme) you believe are needed or not. If, for instance, you’re on the homepage and there’s “Contact Form 7” files loading and you’re sure you’re not using any contact form on the homepage, then you can go ahead and unload the 2 files (CSS & JS) that are loaded by “Contact Form 7”. Same goes for other elements such as sliders, carousels, CSS & JS belonging to Checkout pages for example and so on. There’s no way to automate this 100% as only you (the admin) knows when and how you will use specific plugins (there are ways to add helpers and they will be added later on to the plugin) that will load CSS / JS files. For example, you might have a “Contact” link on the homepage that will open a modal box (that is not seen by default when the page is loading), in this case, you will need to keep “Contact Form 7” files on the homepage.
Often, you don’t need lots of files, I had people telling me that they unloaded sometimes over half of the elements from a page as they were completely useless in that scenario. Moreover, once you unload the “crap” from a page, you can use a cache plugin to minify and combine the remaining loaded files which will result in a lighter and faster website.
I know, it’s not as easy to use other plugins that you just have to enable, but once you understand how this works and clean the right things, your web pages will load much faster. I believe the number of downloads and 5-star reviews this plugin has, tell a lot about its purpose and quality.
One thing I can promise is the fact that I will improve the UI and make it easier to use in the future.
I’m happy to answer any further questions as other people might see this thread and would find this information useful.