I’m using 1.23.16, I’m on auto-update. I also use it on two older sites, for which that checkbox was already checked before it got to this version, and they actually work.
I’ve verified that I can send SMTP and have tried various plugins and methods of using SMTP, so I’m pretty sure it’s related to that checkbox not sticking.
I’ll test again and see if an auto-update fixed it.