I think you're jumping a little ahead on your baby's moods and attachment to your boyfriend - your baby reacts to YOUR moods, not a boyfriend's. If your baby is more active when your boyfriend is about to come home, it's not because the baby predicts your boyfriend is coming home, it's because your heartrate/energy level is elevated.
I say this because it's a concern when parents think their babies know more than they do - and so the parent's expectations are that the baby behave better or socially acceptably.
I feel as though a dad is whoever really cares for, loves, and provides for your child... Especially if the biological father just left you alone because you refused to murder your innocent child. Kerp your head up and thank god that he blessed you with someone that truly loves you and your bundle of joy. It's not about what anyone else says or thinks. It's about your happiness and the well being of your bby.