The recorded text test (RTT) is generally employed to test dialect intelligibility, often to inform language evolution decisions. More than 25 papers employing the RTT method were published on www.sil.org/silesr from January 2009 to March 2013. As introduced by Casad [1974. Dialect Intelligibility Testing. Summer Institute of Linguistics Publications in Linguistics and Related Fields, publication no. 38. Norman: Summer Institute of Linguistics of the University of Oklahoma], the method entails recording a narrative text, developing, and piloting an audio test based on the text, and employing this test with speakers of speech varieties linked to the text’s speech variety. This is the first inquiry to examine the dependability of RTT scores in a generalisable way. This inquiry shows that the selection of a text to be developed into an RTT has notable implications on the results of the RTT which could lead to incorrect recommendations about how well material produced in one speech variety could be employed by speakers of linked speech varieties. The 10 questions Casad (1974, 117) advises may not be adequate to get a stable measure of intelligibility.