I actually made similar changes to mine today, for the purpose of the eventlist.
$out .= '<span class="fc-event-time">';
// always display the start date
$out .= $row->start_date;
// only display the end date if different than the start date
if (!($row->end_date == $row->start_date)) {
$out .= ' - ' . $row->end_date;
}
// only display times if the event is not all-day
if (!$row->allDay) {
$out .= '<br>' . $row->start_time . ' - ' . $row->end_time;
}
It seems to work just fine, though obviously that can change with future versions. It should address all of the possible date and time states
Now one issue was that when displaying both a start-date and an end-date, the line wrapped around. I fixed that by changing to a three-letter display of months in WordPress settings, instead of full-word month names.