SQL
SELECT SUM(p.comment_count) FROM wp_posts p
JOIN wp_term_relationships tr ON tr.object_id = p.ID
JOIN wp_term_taxonomy tt ON tt.term_taxonomy_id = tr.term_taxonomy_id
JOIN wp_terms t ON t.term_id = tt.term_id
WHERE t.name = 'CategoryNameHere'
AND p.post_status = 'publish'
NOTE: Prefix is assumed to be wp_
in the above example.
Wpdb Query
$comments_in_cat = $wpdb->get_var("
SELECT SUM(p.comment_count) FROM $wpdb->posts p
JOIN $wpdb->term_relationships tr ON tr.object_id = p.ID
JOIN $wpdb->term_taxonomy tt ON tt.term_taxonomy_id = tr.term_taxonomy_id
JOIN $wpdb->terms t ON t.term_id = tt.term_id
WHERE t.name = 'CategoryNameHere'
AND p.post_status = 'publish'");
echo $comments_in_cat;
Hope that helps.. ??