Thanks for your answer!
I finally found the reason. On the server of the provider the php variable open_basedir was set so that sys_get_temp_dir could somehow not find the right tmp folder.
As long as I cannot convince them to change this, I changed the lines 124 / 125 in class-wp-smime.php from
$infile = tempnam( sys_get_temp_dir(), 'wp_email_' );
$outfile = tempnam( sys_get_temp_dir(), 'wp_email_' );
to
$tmp_dir = ini_get('upload_tmp_dir') ? ini_get('upload_tmp_dir') : sys_get_temp_dir();
$infile = tempnam( $tmp_dir, 'wp_email_' );
$outfile = tempnam( $tmp_dir, 'wp_email_' );
Of course it is just a temporary solution, but it works for now.