If you defer css, one of two things will happen.
a) Your site will experience a FOUC effect.
You would need to add a critical css code, which depends on your device and window size (it requires coding skills or a developer).
https://tech.namshi.io/blog/2015/04/08/improve-website-page-load-performance-with-critical-css/
b) Even if you add critical CSS and then async CSS, layout will probably still be broken due to css specificity issues.
Basically, with async loading, your css files are not guaranteed to always load in the same order, hence they can rewrite styles and break the design for some users.
CSS should be render blocking and the impact on pagespeed is minimal.
You can also inline it if your css is small enough.
You cannot remove unused css from the files.
FVM doesn’t know which css code is in use, it simply merges the code that is added by your theme or plugins.
