Yes, this is unusual and needs to be looked at. I agree with Annie as to things to check out.
I also think that if he is 3, he is eligible for the public school system to get involved.
What is important is that the sooner the better.
There is something called expressive language disorder, and something called selective mutism, which would be worth reading about just to see if either fits what you're seeing. Also, do you have any indication your child has a hearing issue, or is his hearing fine (like, he turns when there is a loud noise behind him)?
My 2 year old son doesn't talk much either. He makes a lot of sounds, says some words but not a lot. He is #4, and very expressive so everyone knows what he wants without him having to speak. Plus I taught all 4 sign language so he relys more on that that words. I am concerned too but I feel if he is communicating effectively in his own way it's OK. I would be worried if there wasn't any attempt to communicate and had periods of repetition and spacing out.