Website page load speed issue

I have a website hosted on Godaddy Windows server. I have added MH lite theme over it and applied everything related to page speed like WP Super cache, lazy load, image optimization, database optimization but the response time from server or time to first byte is continuously high near about 4-6 secs and whole website takes 10-14 sec to load. I am losing half of my visitors daily as they close the page after 5 seconds.

Now after applying caching with autoptimize plugin the page speed have improved from 14 secs to 3.5 secs according to every web page speed testing site but when I empty site cache and try to open it then it takes more than 14 secs to load the page for the first time. Which means new users will have to wait more than 14 secs to access the page after that caching will take care of it.

Please explain what is happening?