Looks like when the if statements are true, you query_posts twice, duplicating the posts. To get around this move the orderby=rand into the $args array, adjust the second query_posts() to use this array in it’s query, and remove the first query_posts() completely.
So try…
<?php if(of_get_option('frontcat_checkbox') == "1"){ ?>
<?php if(is_front_page()) {
// This sets the values of $args if the above both return true
$args = array(
'cat' => ''.$os_front = of_get_option('front_cat').'',
'post_type' => 'post',
'paged' => ( get_query_var('paged') ? get_query_var('paged') : 1),
'posts_per_page' => ''.$os_fonts = of_get_option('frontnum_select').'',
'orderby' => 'rand');
} ?>
<?php }?>
<?php
// query_posts() occurs once, returning one set of posts matching the values of $args
query_posts($args); ?>
<?php
if(have_posts()):
// If we have the posts, display them ?>
<?php while(have_posts()): ?>
<?php the_post(); ?>
<div <?php post_class(); ?> id="post-<?php the_ID(); ?>">