Scientific research produces new knowledge, technologies and clinical treatments that can lead to enormous returns. Often, the path from basic research to new paradigms and direct impact on society takes time. Precise quantification of scientific output in the short-term is not an easy task but is critical for evaluating scientists, laboratories, departments and institutions. We argue, with others, that current methods are not ideal and suffer from solvable difficulties. Here we propose criteria for a metric to be considered a good index of scientific output. Specifically, we argue that such an index should be quantitative, based on robust data, rapidly updated and retrospective, presented with confidence intervals, normalized by number of contributors, career stage and discipline, impractical to manipulate, and focused on quality over quantity. It should be validated through computational and empirical testing. Given its influence on the efficiency of scientific research, we have a duty to reflect upon and implement novel and rigorous ways of evaluating scientific output. The criteria proposed here provide initial steps towards the systematic development and validation of a metric to evaluate scientific output.