Ignore Terms that Don’t Exist

How do I set the filter to ignore terms that do not exist on a product while filtering? I only want to constrain by active terms.
Some products have dimension terms and particular attribute terms and some have one or the other. If there are instances where something is being filtered by particular attribute terms and all of a sudden products that have dimension only terms disappear is a problem…
Any insight? Thanks!