I agree that you need another opinion unless you can get your doctor to test the Free T4 and Free T3, along with the TSH.
Since you have symptoms of hyperthyroidism, you also should have all three of the antibody tests, even though your TSH is on the higher side. It's entirely possible to have both Graves Disease and Hashimoto's at the same time. For Hashimoto's, you need the Thyroid Peroxidase Antibodies (TPOab) and Thyroglobulin Antibodies (TgAb). Thyroid Stimulating Immunoglobulin (TSI) is the definitive test for Graves Disease.
An ultrasound can help diagnose Hashimoto's and determine if characteristics are indicative of cancer, although a biopsy is the only way to diagnose cancer. It's important to note that less than 5% of all thyroid nodules are cancerous.
Sometimes, nodules can produce hormones independently of the thyroid. This could explain hyper symptoms with a higher TSH. In order to determine if nodules are producing independently of the thyroid, you will need a nuclear medicine scan, otherwise called an uptake scan.
A doctor that's inclined to only test TSH is probably not willing to go out of his way to do other tests necessary to determine whether or not you have a thyroid condition or what is the cause of your symptoms.
Yes, you definitely need a second or third opinion because this doctor appears to be misdiagnosing you. He doesn't quite understand that a goiter means thyroid disease, which thyroid tests are necessary, and that TSH 3.620 means your thyroid has a problem. What kind of symptoms are you having? Are you sure you don't mean hypo (slow), and not hyper (fast)? TSH at 3.620 is getting to be too high, and is associated with hypo symptoms. Unfortunately, that doctor did not do blood work on your thyroid. He only tested TSH, which is pituitary hormone which communicates with your thyroid. TSH does not mean a whole lot by itself. Thyroid blood work are three tests. TSH, FT3, and FT4. The last two are your thyroid hormones. The first one is the pituitary hormone that tells your thyroid how much hormone to produce. With your level of 3.620, your TSH is probably getting close to the top of the lab range. Knowledgeable doctors know that TSH of 3.620 means that the thyroid is slow and is not able to produce adequate thyroid hormones on it's own. Low or high thyroid hormones cause symptoms. In true hyperthyroidism, TSH is usually very low and FT3 and FT4 are usually very high. In the beginning of hypothyroidism when the thyroid first starts to have problems, sometimes people go between weight loss and hyper-like symptoms, and weight gain and hypo-like symptoms until eventually the thyroid can't make enough thyroid hormones and then the person is permanently hypothyroid.
On top of those standard complete thyroid tests, you also need to test your thyroid antibodies TPO/ab and TG/ab to check to see if you have autoimmune thyroid disease.
The fact that you have a swelling in your neck means that you probably have thyroid disease. A thyroid ultrasound is required to determine if you have thyroid nodules and to make sure that they are not cancerous. It is standard for a doctor to palpate your thyroid to make sure it is not swollen. If it is swollen, a thyroid ultrasound must be ordered to determine the exact condition of your thyroid.