This article describes the process of assessing the effectiveness of the operation of automated information systems in colleges using the method of pairwise comparison and discusses numerical representations used with the above method. The author lists methods for the subjective measuring of the effective operation of automated information systems. The article proposes a way to automate the methods that makes it possible to simplify performing calculations and reduce the amount of time it takes to determine the outcome of a specific task. The author provides an algorithm with the results of work carried out.