This paper describes the use of the 'cloze procedure' to make a comparative evaluation of some currently available MT systems. This technique, which involves masking some words in a text, and then asking subjects to guess the missing words, was used in the 1960s and 1970s as an evaluation method, but no more recent use has been reported, even though the methodology is simple to implement and provides an objective result. Here two experiments are reported in which three MT systems were tested against a human translation, with texts from three different genres, to see whether the procedure can be used to rank MT systems against each other. The paper discusses some details of the procedure which provide important variables to the test, notably what percentage of words are masked, and whether the scoring procedure should be right wrong, or should differentiate between different degrees of wrong answer (crediting close synonyms and other plausible near-misses). Other aspects of the procedure which may affect the tests usability are discussed. Especially of interest is the fact that there seems to be a lower quality threshold below which the procedure is less discriminatory: the translation is so bad that the subjects cannot make reasonable guesses at all.