This paper summarizes some results of the first ethnographic research study conducted in Colombia in three different elementary schools for deaf children in which bilingualism (sign-language and written-language) is starting to be emphasized. This study focuses on the teaching of mathematics in classrooms for deaf children using contexts proclaimed as bilingual. The participants in the study were first grade teachers. The analysis presented in this paper illustrates the struggles that teachers experience teaching arithmetic in such a context. The teaching of arithmetic using bilingualism requires three types of semiotic registers: sign-language and written-Spanish, and the Hindu-Arabic numeration system. The analysis indicates some puzzling teaching learning issues interweaving language and mathematics. These issues are of linguistic and communicative, social and cultural, and cognitive and pedagogical nature.