Recent studies have shown that during unimanual motor tasks, aging adults show bilateral recruitment of primary motor cortex (M1), while younger adults show a suppression of the ipsilateral motor cortex. Additional work has indicated that increased bilateral M1 recruitment in older adults may be deleterious when performing some motor tasks. However, higher levels of physical fitness are associated with improved dexterity and fitness may mitigate the loss of both inhibitory and excitatory communication in aging adults. The goal of this study was to assess dexterity and interhemispheric motor communication in physically fit and sedentary middle-age (40–60 years) right handed participants using tests of hand deftness and transcranial magnetic stimulation (TMS). To behaviorally assess the influence of interhemispheric communication on motor performance, participants also perform the coin rotation deftness task while maintaining pinch force with the opposite hand (bimanual condition). We correlated these behavioral measures with the ipsilateral silent period using TMS to assess interhemispheric inhibition. Our results show that the middle-aged adults who were physically fit had better dexterity of their right hand (finger tapping and peg-board). When performing the coin rotation task the fit group had no between hand differences, but the sedentary group’s left hand performance was inferior to the their right hand. We found that better dexterity correlated with ipsilateral silent period duration (greater inhibition) thereby supporting the postulate that fitness improves interhemispheric motor communication.