I had gone up one cup size during pregnancy and another cup size after 3-4 days after my baby was born when my milk came in. It was really obvious all of a sudden that the milk came in.
You don't have anything to worry about even if they don't grow it doesn't determine how much milk you make. They could stay the same size and you have an over supply
And even if they don't go through crazy growth (mine didn't, after my milk regulated I am still the same size as before) it does NOT matter what size they are, you can still produce for baby :)
Mine didn't really change through my first pregnancy but when my milk came in at three days postpartum I went from a 34B to a 36E overnight. It'll happen. Don't worry. And they won't stay quite so crazy either - after the first engorgement passed they were 36Ds and after two years of breastfeeding they settled to 36C.