I've been told I carry small too. It can be very irritating but listen to your doctor. See if you can get an ultrasound to see if baby is growing at the right pace. Or have them measure your uterus to see if it's where it needs to be.
Try not to worry. Your doctor would tell you if you needed to be concerned. We're all different.
Everyone carries differently, I'm still have a small bump and I'm 28 weeks. My mum carried small for all her babies, no one even knew she was pregnant, all her babies were average sized too.
I have a friend who is a couple of weeks ahead of me and she's her bump is much bigger than mine and we were the same size before pregnancy.