Hello @tobbe_l
Thank you for the information. I can see that you have wp super cache installed at the moment.
We cannot help you if W3 Total Cache is not installed because we need to check the behavior.
As I can see in the test results, you need to enable compression which can be done in W3 Total Cache Browser Cache settings.
Also, the bottleneck of your website performance is your images, for example, https://spwc.se/wp-content/uploads/2018/11/ettapuff_london19.jpg
which in some cases may take up to 20s to load according to the test results.
What I would advise you to do is to completely remove the other caching plugin, performed test on GtMetrix, and then install W3 Total Cache, enable Page Cache and Browser Cache in General settings and in Performance>Browser Cache enable Set Expires header, Cache-Control header, Etag, and Gzip compression in the General sub-box, save all settings and purge the cache.
Perform another test on Gtmetrix (Please note that just after purging the cache it may take some time so take a couple of tests) and see the improvements.
Thank you!