The accentedness judgment task is widely used in the study of second language speech, and has proven to elicit remarkably reliable ratings across listeners. Despite this reliability, however, little is known about how listeners arrive at accentedness ratings. The present study seeks to illuminate the role that listener attitudes and expectations play in assessments of accentedness by probing listeners’ explicit criteria for the judgments. The study asked ten native English listeners to rate the accentedness of five native Bosnian and five native English speakers, and then to justify their ratings in a semi-structured interview. Analysis of the interview data indicates that despite the elegance of quantitative accentedness data and its remarkable reliability, it appears that when making accentedness judgments, listeners may activate a complex set of attitudes and expectations about speakers that go far beyond a straightforward bottom-up analysis of the speech signal.