do not have sufficient permissions

This really is a known problem and there are multiple other Support incidents open about this exact problem. If you are using Multi-site then you need to do all of the Identity Provider settings in the Network Admin Settings and the General and Service Provider settings in the Site Settings.

Also you will need to go into the saml directories under each subsite and add the correct IdP information.