Have you ever heard of PANDAS? It is Pediatric Autoimmune Disorder Associated with Strep - or PITAND? Have you noticed these symptoms come on after a viral or bacterial infection? Disease is characterized by overnight onset of tics/OCD/Tourettes. Treatment can involve antibiotic or steroid - or in serious cases IVIG. I would research this.
I would consider 2 things: One is maybe he is stressed about something - school, bulling, to much homework, and this is the only way he knows how to release stress. Second thing would be to make an appointment with his pediatrician and then (possibly) a therapist. He could simply be going through a growth spurt and have some muscle tension in his hands or wrists or maybe has some mild muscle spasms. Then if the pediatrician can find nothing wrong or nothing to contribute it too, a therapist can often get our kids to talk to them more than they will talk to us and can address any problems and will speak with you about any concerns they have regarding your child's habits.
First of all, I don't know what is causing your son's problem. You really should take him to his doctor and mention your brother's past and present condition. I was wondering what kind of mental problems your brother seems to have. Is it possible your brother is on the autistic spectrum? Maybe something like Asperger's syndrome which is high functioning autism - they can display tics and twitches and OCD (obsessive compulsive disorder) type symptoms along with many others. Many people with Asperger's go undiagnosed or misdiagnosed.
I hope this hellps. My middle daughter always has to stretch her back and her limbs and has a tendency to get irritable very easily. She was like that as a baby too. She always looked uncomfortable in her own skin. She actually has several other issues. I read about sensory integrative disorder and she pretty much fits that diagnosis. I recently started her on a gluten free diet for other reasons and I noticed her irritability has decreased about 80%. I asked her if she still feels the need to stretch and she said a little bit sometimes, but not as bad as before. I mention this because even though we have genetic predispositions, environment (food, air, etc.) can set them off. Maybe your family lacks the ability to break down certain foods like gluten f(which is in wheat, barley, rye and possibly oats) and/or cow's milk products or just a general carbohydrate intolerace. These intolerances, etc. cause vitamin, mineral, and other nutritional deficiencies. Gluten is known for neurological side effects (as I'm sure other foods are).
I might be way off base, but food elimination is something to try - maybe he will have a reduction in symptoms like my daughter.....or maybe not.
Hang in there! I know it's not easy, but early diagnosis of his problem might change his outcome in the long run.