Introduction: Multiple choice questions (MCQs) are widely used to assess medical residents. The study aims to analyse MCQ exams of medical residents administered at Kermanshah University of Medical Sciences during 2008-2012. Method: Data of this retrospective study were extracted from 63 MCQ exams including 9600 tests. Quantity related variables included discrimination index, difficulty level, reliability of test index (KR20). Quality related variables included proportion of questions without structural defect and proportion of taxonomy 1, 2 and 3 questions. Data were collected using MCQs from question bank and analyzed by SPSS V.16 software. Results: Over the specified five years, the average difficulty level was %0.62, the average discrimination index was 0.27 and the average of reliability coefficient (KR20) in total exams was 0.88. The average of questions in Taxonomy 1 over the specified years was 33.5% and questions with taxonomy 2 and 3 were66.5%. The average of questions without structural defect was 62.6% which fell in the unacceptable range. A significant difference was observed in reliability coefficient (KR20)- (P=0.017), difficulty level (P=0.001), taxonomy 1 (P=0.001) and Taxonomy 2 and 3 (P=0.006). Conclusion: According to findings of this study, taxonomy 2 and 3 were decreased and taxonomy 1 was increased over the specified five years. We recommend the specialist board members and the Deputy for Education of the Ministry of Health to consider the results of this study and react to them by improving MCQ quality of future exams.