Defer Google Tag Manager

Looking at the code I see that the async is added but I was wondering if it is possible to enqueue the google tag in defer mode without use javascript.
In this way should be more fast and wait that everything else is downloaded.