Back Up/Saving Site

I’m a newbie to WordPress, but compared to Joomla and Drupal, I love it. How do go about backing-up my new website, just in case I screw it up with a plug-in or update? Do I save every web page to a flash drive or something like that or is there way to do it using WordPress itself or my server on GoDaddy.com?

Thanks for the help.