I just checked my prescription bottle and see the calcium channel blocker is Diltiazem 120 mg twice a day. This is a generic medication for the drug you mention, I believe. It is normal release, so I take it every 12 hours. I do believe the slow release isn't available in generic.
I have permanent AFib and take 50 mg Metoprolol ER and 225 mg of (I believe it is the same) channel blocker twice a day. I have no side effects other than some fatigue and general tiredness. Some of this comes from my age and the fact I no longer do serious aerobic exercise. I used to jog 3 miles 3 times a week when on a beta blocker and when on Propafenone which were back then holding me in normal sinus rhythm. With AFib my heart inefficiencies makes it hard for me to walk up hill, forget jogging, so I don't. The combination of BB and CCB keeps my resting HR at bout 80, it is (or was) about 130 with lower dose of only a BB. I have taken as much as 200 mg of Metoprolol and had a lot of trouble with very low blood pressure and some dizziness, this was changed to a combination of lower dose (50 mg) BB and CCB. My blood pressure is near normal and my HR is (resting) is at a reasonable level. I am not a candidate for ablation because of an enlarged heart (left atrium in particular).
Have you discussed an ablation with your doctor? Given troublesome symptoms I think a ablation should be considered.
I think an anticoagulant is standard and I am surprised you have been taking one all along. I take 5 mg of Warfarin each day and a low dose aspirin once a day. I take the aspirin in the morning and the warfarin at bed time.