If they're going only by ultrasound then it's usually at least 20 weeks. Even then, they can be wrong because of things like you mentioned. .. small genitals etc.
There are a couple of different blood tests that can determine gender as early as 10 weeks though. I did the materni-t21 and found out it's another girl so my u/s will be just to see her development.
18 to 24 weeks my obgyn will not even test until 20 weeks but I am sneaking in at 18 bc my next appointment is one week after I find out