Recent studies have suggested the importance of the neighborhood environment in determining the specific type of physical activity. However, few studies on this topic have been undertaken in Japan. This study examined the association of three types of physical activity and their associations with individual and neighborhood environmental factors among middle-aged and elderly Japanese. Participants were 2,449 adults aged 40–69 living in Fujisawa city who had undergone health checkups and responded to our survey by mail. Individual factors, the International Physical Activity Questionnaire (long form), and its environmental module acted as inputs to the study. The adjusted odds ratios (ORs) of high levels of moderate-to-vigorous intensity leisure-time physical activity (LTPA), walking for active recreation, and transportation were calculated in relation to individual and neighborhood environmental factors through multiple logistic regression models. Not working and good self-rated health were significantly associated with a higher level of each physical activity outcome. According to the adjusted ORs, higher educational attainment, higher economic status, good access to exercise facilities, and owning motor vehicles were associated with longer LTPA time. However, different sets of factors were associated with longer walking times for recreation and transportation. The results suggest that diverse individual and neighborhood environmental characteristics are associated with different physical activity outcomes. Therefore, customizing environments to become activity-friendly is necessary to increase physical activity effectively among middle-aged and elderly Japanese.