Unless the user also has your website locally, they won’t be able to access the site to listen to the audio if they don’t have an internet connection, so couldn’t access the audio in the first place!
By default if you include an audio file and use the inbuilt player, it will do something called progressive download, where your browser downloads the file in chunks, so if you have a brief internet outage the player will continue, if the file is downloaded slowly or your connection drops entirely then it will stop or buffer. This is also a quite efficient way of conserving bandwidth for you as the website owner, as the chunks of data are only a few seconds ahead of where the visitor is in the file.
However if you want a visitor to be able to access the file on their own computer, you can link to the file directly. If you uploaded it through the media manager, there will be a direct link to the audio file and you can include this link in your posts (perhaps directly under the player) and the user can download the file and play using their computers own player. The downside to this is they have to download the entire file first normally before they can play the audio, so if they are on slow connection this may take some time.
There are a few other ways to distribute audio files, for example if you are setting up a podcast, then you can include the audio file in your RSS feed and then people using podcasting software or even software like iTunes will be able to access the audio directly through their software.
