Exclude post categories from sitemap

I have two post categories, ‘red’ and ‘blue’, subcategories of ‘color’, that I wish to exclude from post sitemap. I tried the following but it does NOT work:

if( ! function_exists('custom_sitemap_exclude_taxonomy') ){
	function custom_sitemap_exclude_taxonomy( $value, $taxonomy ) {
		$taxonomy_to_exclude = array('red','blue');
		if( in_array( $taxonomy, $taxonomy_to_exclude ) ) 
			return true;
	}
	add_filter( 'wpseo_sitemap_exclude_taxonomy', 'custom_sitemap_exclude_taxonomy', 10, 2 );
}

I also tried the following

$taxonomy_to_exclude = array('color/red','color/blue');

but it does not work too. What did I wrong?