Whether gallery thumbnails — in edit or live — are square or proportional is based on Settings > Media > Thumbnail size > Crop thumbnail to exact dimensions (normally thumbnails are proportional).
At the time of image upload, WordPress will create the thumbnail based on this setting. If you change the setting, you need to regenerate thumbnails so that new thumbnails are created based on the new setting.
Or maybe I’m misunderstanding the question. Are you instead talking about the “Insert Media” thumbnails you see when you click “Add Media” when editing a post? The thumbnail grid of images in your media library are square. That’s just the way WordPress is.
The thumbnails in Media > Library will display square or proportional based on your Settings > Media > Thumbnail size > Crop setting.