Human evaluations of machine translation are extensive but expensive. Human evaluations can take months to finish and involve human labour that can not be reused. The authors of this article propose a method of automatic machine translation evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evaluation, and that has little marginal cost per run. They present this method as an automated understudy to skilled human judges which substitutes for them when there is need for quick or frequent evaluations.