Website speed highly variable

I have speed tested my site extensively and it ranges in speed from 20s to 500ms, the longer time being usually at the beginning of testing but not always. My site has a low visitor volume.

I have set up wp supercache, auotoptimise and ewww image optimiser and have cdn set up (probably not necessary but I am trying everything). I have tried deactivating all the plugins and theme and it is still randomly slow.

The site now regularly scores high in performance (Pingdom 95%, PageSpeed 99%, YSlow 95%) even when the site is slow. Pagespeed regularly sites server response time as an issue and I know that doesn’t always mean it is the server’s fault but I think I have done everything I can on my side. TSO host (UK) has done a few things – gzip compression, optimisation – but says it can do no more.

If I have optimised the site as much as I can and I have such a high performance grade would it be correct to say that the issue is with my shared server?

The site is onebluepebble.co.uk

Any insight very much appreciated