Remove unnecessary fields from the Select a Feild

I have a number of fields for my members. I had assumed that the Select a Field box would only include the fields that were selected as “Add to Filters” would show up in that box.

It appears all fields show up in the select a fields box. Some are supposed to be hidden to normal users, some shouldn”t be in the list at all because they just clutter it up.

How can I customize what shows up in the “Select a Field” on the group page?