Models of communicative ability in a second language invoked in determining the construct of broadly employed tests of communicative language competence have drawn mainly on the work of language specialists. The risk of exclusive dependence on language expertise to conceptualize, design and manage language tests is that test scores may convey meanings that are misaligned with the values of non-language specialists, that is, those without language expertise but perhaps with expert knowledge in the domain of interest. Neglect of the viewpoint of lay (i.e., non-linguistic) judges on language and communication is a grave validity concern, since they are the ultimate arbiters of what counts for efficient communication in the pertinent context of language use. The paper reports on three research studies exploring the validity of rating scales used to assess speaking performance on a number of high-stakes English-language tests developed for professional or general proficiency assessment purposes in Korea, Australia, China, and the UK. Drawing on Jacoby and McNamara's (1999) notion of “indigenous assessment”, each project attempted to identify the values underlying non-language specialists' judgements of spoken communication as they rated test performance or participated in focus-group workshops where they viewed and commented on video- or audio-recorded samples of performance in the relevant real-world domain. The findings of these studies raise the question of whether language can or should be assessed as object independently of the content which it conveys or without regard for the goal and context of the communication. The studies' findings also cast doubt on the notion that the native speaker should always serve as benchmark for judging communicative effectiveness, especially with tests of language for specific purposes, where native speakers and second-language learners alike may lack the requisite skills for the kind of effective interaction demanded by the context.