It has long been known that the identification of aural stimuli as speech is context-dependent (Remez, Rubin, Pisoni, & Carrell, 1981). Here, we demonstrate that the discrimination of speech stimuli from their nonspeech transforms is further modulated by their linguistic structure. We gauge the effect of phonological structure on discrimination across different manifestations of well-formedness in two distinct languages. One case examines the restrictions on English syllables (e.g., the well-formed melif vs. ill-formed mlif); another investigates the constraints on Hebrew stems by comparing ill-formed AAB stems (e.g., TiTuG) with well-formed ABB and ABC controls (e.g., GiTuT, MiGuS). In both cases, nonspeech stimuli that conform to well-formed structures are harder to discriminate from speech than stimuli that conform to ill-formed structures. Auxiliary experiments rule out alternative acoustic explanations for this phenomenon. In English, we show that acoustic manipulations that mimic the mlif-melif contrast do not impair the classification of nonspeech stimuli whose structure is well-formed (i.e., disyllables with phonetically short vs. long tonic vowels). Similarly, nonspeech stimuli that are ill-formed in Hebrew present no difficulties to English speakers. Thus, nonspeech stimuli are harder to classify only when they are well-formed in the participants’ native language. We conclude that the classification of nonspeech stimuli is modulated by their linguistic structure: inputs that support well-formed outputs are more readily classified as speech.