admin-ajax.php / wp-load.php loads too many features hence causes slow response

I noticed that wp-load.php / admin-ajax.php loads almost all of wordpress features. This would take naturally around 500ms to 1500ms of responds time after executing any custom scripts. Even the REST API has the same issue. I personally use autoload to load classess in my projects hence my systems only require_once scripts that are needed or used by the page. Perhaps this could reduce the bloat that is caused by wp-load.php / admin-ajax.php.