Custom login and registration form

For custom login form, see Customizing the Login Form. For registration, review the pertinent portions of wp-login.php. There may be action hooks you can use for this, similar to the login form. If you need to create your own page, manage the form submit in the same manner as wp-login.php. Same goes for error handling, use existing source code as guidance on what to do.

Another option is to seek out other plugins that offer custom forms and see what they do.