When we Add the defer attribute to JavaScript files, the wp-admin breaks.

Console logs
Uncaught ReferenceError: wp is not defined
Uncaught ReferenceError: acf is not defined
Uncaught ReferenceError: tinymce is not defined
Uncaught TypeError: a is undefined
etc etc

I don’t think it’s possible to defer js files in backend. Perhaps you should only check for frontend.