WP Media Uploader Unselect Image on close

Hello,
I have created an image uploader on Custom Taxonomy [like a featured image for taxonomy]. I am able to upload the image and save it. The problem is, if I do not select image from media uploader popup and just try to Save it then the image selected in previous taxonomy term gets added on the current taxonomy term as well.

Lets say I have selected an image for taxonomy term A and save it, it gets saved. Again if I add another taxonomy term B without selecting the image then the image added on taxonomy term A gets saved on the taxonomy term B as well. The media uploader keeps the previously added image as selected even after closing it.

Below is my code:

jQuery(document).ready( function($) {     
        $('.ct_tax_media_button.button').click(function(e) {
          e.preventDefault();
          var uploaded_image = '';
          var image_url = '';
          var image = wp.media({ 
            title: 'Upload Image',
            multiple: false
          }).open()
          .on('select', function(e){
            var uploaded_image = image.state().get('selection').first();
            var image_url = uploaded_image.toJSON().url;
              $('#category-image-id').val(uploaded_image.toJSON().id);
              $('#category-image-wrapper').html('<img class="custom_media_image" src="" style="margin:0;padding:0;max-height:100px;float:none;" />');
              $('#category-image-wrapper .custom_media_image').attr('src',image_url).css('display','block');
          });
        });

     $('body').on('click','.ct_tax_media_remove',function(){
       $('#category-image-id').val('');
       $('#category-image-wrapper').html('<img class="custom_media_image" src="" style="margin:0;padding:0;max-height:100px;float:none;" />');
     });
     // Thanks: https://stackoverflow.com/questions/15281995/wordpress-create-category-ajax-response
     $(document).ajaxComplete(function(event, xhr, settings) {
       var queryStringArr = settings.data.split('&');
       if( $.inArray('action=add-tag', queryStringArr) !== -1 ){
         var xml = xhr.responseXML;
         $response = $(xml).find('term_id').text();
         if($response!=""){
           // Clear the thumb image
           $('#category-image-wrapper').html('');
         }
       }
     });
   });  

Any help on this will be highly appreciated. Thanks.