After further experimentation, I have concluded that I misunderstood the meaning of restricting access to media. It only determines whether or not the media item will be displayed on a given page, not whether or not it can be accessed directly given the file url.