Over the past decade, several methods have been used to compare the performance of fire danger indices in an effort to find the most appropriate indices for particular regions or circumstances. Various authors have proposed comparators and demonstrated different responses of indices to their tests, but rarely has much effort been put into demonstrating the validity of the comparators themselves. We present a demonstration that many of the published comparators are sensitive to the different frequency distributions, that may be inherent in the performance of the different indices, and outline a non-parametric method that may be useful for future work. We compare four hypothetical fire danger indices, three of which are simple mathematical transformations of each other. The hypothesis tested is that the comparators often used in such studies may indicate spurious performance differences between these indices, which is found to be the case. Non-parametric methods are robust to differences in index value frequency distribution and may allow more valid comparisons of fire danger indices. The new comparison method is shown to have advantages over other non-parametric comparators.