Response length, expression range and content, and native speaker evaluations of Japanese EFL students' responses to production questionnaires and role plays are compared. In this exploratory study, it is concluded that production questionnaire scores do not allow substitution for role play scores.