Found a way around this for now. Not sure if it is the proper way to do it but it works so far.
I commented out the add action in the plugin file for putting it on the product page and commented out the lines under get tax label that mentioned the attribute label to remove the attribute category from the other areas.