?Deactivated plugin insecure?

I had an unusual request from the personal who is hosting my site.

I found one plugin that caused a successful hack attack few days ago.
After I solved the problem I disabled it.

The technical supports is saying that is not enough.
In his personal words:

“Unfortunately disabling the plugin won’t be enough as the files are still there and can easily be accessed by anyone just by browsing your plugins folder.
At this point I would recommend replacing the plugin with another one, preferably one that is updated often so that these issues do not reoccur.”

Is this true? Is there a way that a disabled plugin represent a threat?
Thanks you