How long has this been going on? These symptoms are often associated with hoarseness and that is often associated with thickening of the vocal cords. This can happen due to inflammation caused by various things. Chemical irritants is common (smoking) even if you smoke infrequently. Other things can cause it too such as gastrointestinal reflux (gerd). Gerd can be such that you also don't know it is going on (silent). So, this is something to talk to a doctor about in general. I think I'd go ahead and get evaluated.
This article had a lot of information in it. It talks about a 'breathy' voice being common if someone has developed a polyp on their vocal cord. A benign tumor or cancer (don't worry or panic, really, anything that happens to us seems like it 'could' be cancer and rarely is) are other reasons. A raspy voice is more indicative of vocal cord thickening is due to irritation. A shaky, breathy voice is indicative of not getting enough air to push the sound out which sounds like it also could be the situation with you. That last issue does sound to have similarities as you describe. Then you have to work with your doctor to determine WHY. https://my.clevelandclinic.org/health/diseases/17105-hoarseness-frequently-asked-questions
Probably all doctors will tell you for now to stop smoking entirely and to get plenty of fluids (extra).
Hello~I would certainly see your GP about this issue, if for no other reason, peace of mind. I agree with Special Mom, the doctor, (and I agree) will tell you to stop smoking, there are just too many negative side effects to it, plus, you will save some money by stopping.