He is trying out this fantasy, that is for sure, possibly in what he feels is a safe place to say the words out loud. I suppose the question is whether it's ever been any more than a fantasy or ever would be more than a fantasy. If this were my husband, it would bother me a lot. I would feel like he was enlisting me to abet his desires and that those desires are pretty creepy.
(I would not want to have a daughter with this man, either.)
Most definitely sounds to me like he's attracted to little girls and especially because it's escalating (getting younger and younger) this would be a complete turn off for me - it's hard to unring a bell, - and it would be really hard to think that as i was getting older, his preference is getting younger. Did he start this after you got pregnant or before?
That's a red flag for me. But also because I was sexually molested as a child by my own sperm donor.