Little is known about the role of perceived neighborhood environment as a determinant of physical activity (PA) and sedentary time (ST) in understanding obesity-related health behaviors. We focus on a biracial, socioeconomically diverse population using objectively measured ST, which is under-represented in the literature. We examined the association between self-reported neighborhood perception (Likert-scale questions), PA using the Baecke questionnaire, and both non-sedentary time and ST using accelerometry from wave 4 of the Healthy Aging in Neighborhoods of Diversity across the Life Span (HANDLS) study ( = 2,167). After applying exclusion criteria, the sample size was = 1,359 for analyses of self-reported PA and = 404 for analyses of accelerometry data. Factor analysis identified key neighborhood characteristics to develop a total neighborhood perception score (NPS). Higher NPS indicated less favorable neighborhood perception. Linear regression was used to determine the relationship between NPS, PA, non-sedentary time, and ST. Complete data were available for = 1,359 [age 56.6(9.0) years, 59.5% female, 62.2% African American] for whom we identified four neighborhood perception factors: (1) concern about crime, (2) physical environment, (3) location of violent crime, and (4) social environment. Worsening perception of the overall neighborhood [β = -0.13 (SE = 0.03); = 0.001], the physical environment [-0.11 (0.05); = 0.03], and the social environment [-0.46 (0.07); < 0.0001] were associated with decreased PA. Worsening perception of the overall neighborhood [1.14 (0.49); = 0.02] and neighborhood social environment [3.59 (1.18); = 0.003] were associated with increased ST over the day. There were no interactions for race, sex, poverty status, or economic index. Poor overall neighborhood perception, perceived social environment, and perceived neighborhood physical environment are associated with PA and ST in a multi-racial, socioeconomically diverse cohort of urban adults. The HANDLS study is registered at ClinicalTrials.gov as NCT01323322.