Amphtml head tag missing with SuperCache

We have just setup AMPforWP and when surfing direct to the /amp/ version of the pages, we are seeing the AMP page.

When logged in the WordPress interface the amphtml tag is present:
<link rel=”amphtml” href=”https://siteurl/amp/”&gt;

However when I log out from admin, the amphtml tag is missing and hence the site is not getting registered as AMP enable.

As the pages are server with WP Super Cache (through Nginx), I’ve tried switching on WP Super Cache ‘Cache HTTP headers with page content.’ and completely switching it off without any change. The theme has a normal <?php wp_head(); ?> function through which I guess the amphtml tag needs to be added.

What could be the problem?

Andre