Offline page not cached?

I’ve installed on 2 sites.

When you switch the internet off, I get the homepage rather than the offline page.

I’ve created a page on each site and selected it in the options, but it never shows when a user is offline.

Does it only work if the user has the offline page cached already??