CPT and Taxonomy rewrite rules conflict.

I am trying to display the hierarchy with my CPT and Taxonomy and I can only get one or the other to work. My permalink is displaying correctly but I either get a 404 for the taxonomy page or the post page. I know it can be done because the default WordPress “Categories” works exactly the way I want this to. How can I isolate my rewrite rules so I can get both instead of one or the other?

$newRules  = array(
	'cpt-slug/(.+)/(.+)/(.+)/?$' => 'index.php?cpt-slug=$matches[3]',
	'cpt-slug/(.+)/(.+)/?$'      => 'index.php?cpt-slug=$matches[2]',
	'cpt-slug/(.+)/?$'           => 'index.php?cpt-slug=$matches[1]',
	'cpt-slug/?$'                => 'index.php?cpt-slug=$matches[0]',
	'cpt-slug/(.+)/(.+)/(.+)/?$' => 'index.php?tax-slug=$matches[3]',
	'cpt-slug/(.+)/(.+)/?$'      => 'index.php?tax-slug=$matches[2]',
	'cpt-slug/(.+)/?$'           => 'index.php?tax-slug=$matches[1]',
	'cpt-slug/?$'                => 'index.php?tax-slug=$matches[0]',
);