[Plugin: WordPress SEO by Yoast] Where to set separator character?

In yoast titles settings you can just replace %%sep%% with the character you want to use

|
>
–

etc. Try that.

In your WordPress template put your desired sep (separator) in the wp_title() tag. So if you want to use the pipe | character, your wp_title should look like:

wp_title('|')

This is usually in the head.php file of your theme. Most likely between the <title></title> HTML tag.

For my themes, I use the following PHP which can be used conditionally with or without SEO by Yoast:

<title><?php
	### Print the <title> tag based on what is being viewed.
	global $page, $paged;

	if(defined('WPSEO_URL')){ // If Using WordPress SEO Yoast - let it override
		wp_title('|');
	} else {
		wp_title( '|', true, 'right' );
		// Add the blog name.
		bloginfo( 'name' );
		// Add the blog description for the home/front page.
		$site_description = get_bloginfo( 'description', 'display' );
		if ( $site_description && ( is_home() || is_front_page() ) ) {
			echo " | $site_description";
		}
	}
	// Add a page number if necessary:
	if ( $paged >= 2 || $page >= 2 ) {
		echo ' | ' . sprintf( __( 'Page %s', 'Test Theme' ), max( $paged, $page ) );
	}
	?></title>

The above will allow you to use the %%sep%% designation in SEO by Yoast and use your own separator character.