This study sets out to examine whether ESL teachers design evaluation criteria according to the demands of different question types assessing literary competence. Four students enrolled in an MA TESL programme were asked to respond to three types of questions – a summary, a content based question, and a reader response question - based on one literary text. Five ESL teachers were asked to select appropriate evaluation criteria for each question type and assess student performance. The results show that all the teachers differentiate between content knowledge and language competence in identifying evaluation criteria for the three questions. But most of them do not select criteria to reflect variations in demands made by each question type. A pedagogical implication of the study is that ESL teachers need training to design evaluation criteria that can match the demands of different question types. This would make content based assessment fair, ensure high inter-rater reliability and produce beneficial washback for ESL students.