Server speed problem what can be issue

Hi guys this is my site of cricket and i checked the speed insights by google and got speed insight result then what i should conclude from server response time . Is this my site issue or any other problem with hosting or cms . I just want any one to help me on this and I’ve another issue that when i disable my plugin wp super cache it happens and when i activate caching it’s response time is increased. what i should do ? i just don’t want to enable cache plugin because my SEO SEARCH TERMS TAGGING don’t work and also my KENTO POSTVIEWS COUNTER don’t work . I’m stuck in this problem now suggest me server changing or extra server or what should i do. I want to resolve this issue