Placing this on top of the .htaccess file resolved the problem. This forwards http to https when there is a proxy involved.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP:X-Forwarded-Proto} !https
RewriteRule .* https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
</IfModule>
I think the plugin should have a force SSL option.