Permalinks clashing with Directory Structure.

Is there a way to modify wordpress or the htaccess file to skip certain directories?

I use wordpress permalinks on my website, and as I understand it, any attempt to access a directory on the server is redirected to the index.php on the root, which lets say is https://www.servername.com

The problem I’m having is that I have a directory https://www.servername.com/downloads/ , which has its own index file and is seperate from the wordpress installation.

There is no downloads category or tag or any other such wordpress link to the directory from the main site. And yet, every time I try to access ‘downloads’, it takes me to a 404 page generated by wordpress.

Thanks!