In this chapter the authors propose a method for translation assessment that implies a rupture with traditional methods where the evaluator judges translation quality according to a series of pre-established criteria. The Calibration of Dichotomous Items (CDI) is a method for evaluating translation competence that essentially transfers the well-known “item”-concept from language testing theory and practice to translation assessment. They present the results of a controlled empirical design in which three evaluation methods are compared with each other: a holistic (intuitive-impressionistic) method, an analytical method that makes use of assessment grids, and the calibration of CDI-method. The central research question focuses on the reliability of these methods with reference to each other.