This is sorted!
Steps I followed.
Go to Product>Attributes then click edit on the size attribute of your choice. Once on the edit screen change the default order type to custom ordering.
Then go back to Products>Attributes and click on Configure Items and drag the items in the order you want them to display.
And done.