Colina (2008) proposes a componential-functionalist approach to translation quality evaluation and reports on the results of a pilot test of a tool designed according to that approach. The results show good inter-rater reliability and justify further testing. The current article presents an experiment designed to test the approach and tool. Data was collected during two rounds of testing. A total of 30 raters, consisting of Spanish, Chinese and Russian translators and teachers, were asked to rate 4–5 translated texts (depending on the language). Results show that the tool exhibits good inter-rater reliability for all language groups and texts except Russian and suggest that the low reliability of the Russian raters’ scores is unrelated to the tool itself. The findings are in line with those of Colina (2008).