This study determined whether there is an association between social capital and a composite outcome of decline in Activities of Daily Living (ADL) and death by gender. A prospective 3.5 year cohort study was conducted in a rural town in Japan. The study participants were 984 individuals aged 65 years and older with not impaired on ADL at 2010 baseline survey. Social participation and generalized trust were measured as social capital. The individual level responses were dichotomized and aggregated into the community level (eight areas). Multilevel logistic regression adjusting for covariates revealed that social participation at the individual level was significantly associated with higher odds of composite outcome (OR of “not participate” = 1.97, 95% CI = 1.38–2.81). Regarding generalized trust, only in men, there was an inverse association at the community level (OR of “low” = 0.55, 95% CI = 0.32–0.96), and a positive association at the individual level (OR of “tend to be careful” = 2.22, 95% CI = 1.27–3.90). These results suggest that social capital were associated with a decline in ADL and death and that the association may differ by gender.