Well, definitely the more likely candidate is the guy you had relations with at the end of December.
That sounds pretty reliable. It would be unusual to be 4 weeks further along and not have that caught at an ultrasound performed around 16 weeks.
How was the estimation of your due date calculated? Did you (or a doctor) just calculate from the first day of your last regular period, or did the due date result from a sonogram or lab work?