Measurement and improvement of e-service quality is important for sustaining competitive advantage of an e-commerce web site. A number of e-service quality measures have recently been developed that predominantly rely on the subjective evaluation of diverse quality attributes. To increase the objectivity of e-service quality measurement a recently developed measure of e-service quality has been adapted in a way that replaces its Likert type scales for evaluation of specific attributes with check-lists that include specific observable elements for appraisal and a scoring procedure. The consistency of scoring with this adaptation of an e-service quality measure was investigated in comparison to the use of the standard measure with Likert type scales and by the use of untrained and trained evaluators. The results of the empirical evaluation indicate a greater inter-rater consistency when the adapted/objective measure is used for evaluation of the efficiency dimension of e-service quality.