Cover not available

Article In: Applied Pragmatics: Online-First Articles

Exploring pragmatic competence in “New Italians”

Refusals and cancellations in Certit B1 candidates’ voice messages

ORCiD logo with link | University of Innsbruck
ORCiD logo with link | Roma Tre University
ORCiD logo with link | Roma Tre University
This content is being prepared for publication; it may be subject to changes.
Abstract
This study examines the pragmatic competence of L2 speakers of Italian (N = 101) who successfully passed the Certit B1 exam, focusing on their oral production of refusals and cancellations in voice messages. The data, collected from authentic exam performances, were analyzed through two complementary approaches: an analysis based on pragmatic categories identified in previous work and an evaluation by four non-expert L1 raters. The messages were annotated using a mixed human–LLM procedure, allowing for fine-grained coding of pragmatic categories. Results show that L2 speakers tend to produce long and elaborated messages, often characterized by redundancy and extensive use of greetings and address terms. While raters tolerated diverse realization strategies, they penalized incoherence and ambiguity, particularly in terms of unclear illocutionary force. The findings highlight the need for assessment models sensitive to multilingual pragmatics and for pedagogical practices that promote coherence and efficiency in L2 speech act performance.
Article outline
  • 1.Introduction
  • 2.Background
    • 2.1Challenges in the assessment of L2-speakers’ pragmatic competence
    • 2.2Refusing and canceling in a second language
  • 3.Aims and method
    • 3.1Research questions
    • 3.2Participants and context
    • 3.3Data collection instrument and procedures
    • 3.4Annotation of data
      • 3.4.1The coding procedure
    • 3.5The rater-based evaluation
  • 4.Results
    • 4.1RQ1: The L2 speech acts’ structure
      • 4.1.1Refusals
      • 4.1.2Cancellations
    • 4.2RQ2: The rater-based evaluation
  • 5.Discussion
  • 6.Conclusion
  • Supplementary materials
  • Credit contribution statement
  • References
References (51)
References
Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice: Designing and developing useful language tests. Oxford University Press.Google Scholar logo with link to Google Scholar
Bardovi-Harlig, K., & Dörnyei, Z. (1998). Do language learners recognize pragmatic violations? Pragmatic vs. grammatical awareness in instructed L2 learning. TESOL Quarterly, 32(2), 233–259. Google Scholar logo with link to Google Scholar
Bardovi-Harlig, K., & Hartford, B. S. (1991). Saying “no” in English: Native and nonnative rejections. Pragmatics and Language Learning: Monograph Series, 21, 41–57.Google Scholar logo with link to Google Scholar
Beebe, L. M., Takahashi, T., & Uliss-Weltz, R. (1990). Pragmatic transfer in ESL refusals. In R. C. Scarcella, E. Anderson, & S. D. Krashen (Eds.), Developing communicative competence in a second language (pp. 55–73). Newbury House.Google Scholar logo with link to Google Scholar
Bella, S. (2011). Mitigation and politeness in Greek invitation refusals: Effects of length of residence in the target community and intensity of interaction on non-native speakers’ performance. Journal of Pragmatics, 43(6), 1718–1740. Google Scholar logo with link to Google Scholar
(2012). Pragmatic development in a foreign language: A study of Greek FL refusals. Journal of Pragmatics, 44(13), 1917–1947. Google Scholar logo with link to Google Scholar
(2014). Developing the ability to refuse: A cross-sectional study of Greek FL refusals. Journal of Pragmatics, 611, 35–62. Google Scholar logo with link to Google Scholar
Bianco, A., Brocca, N., & Garassino, D. (2026). Assessing the potential of LLM-assisted annotation for corpus pragmatics: The case of humor. Corpus Pragmatics, 101, Article 33. Google Scholar logo with link to Google Scholar
Bley-Vroman, R. (1983). The comparative fallacy in interlanguage studies: The case of systematicity. Language Learning, 33(1), 1–17. Google Scholar logo with link to Google Scholar
Brocca, N. (2021). LADDER: Un corpus di scritture digitali per l’insegnamento della pragmatica in L2. Un esempio di analisi di disdette in WhatsApp. Italiano LinguaDue, 13(1), 241–259.Google Scholar logo with link to Google Scholar
(2024). LADDER: A corpus for pragmatic competences in Italian L1/L2 [Data set]. [URL]
Brocca, N., Nuzzo, E., Cortés Velásquez, D., & Rudigier, M. (2023). Linguistic politeness across Austria and Italy: Backing out of an invitation with an instant message. Journal of Pragmatics, 2091, 56–70. Google Scholar logo with link to Google Scholar
Brocca, N., Nuzzo, E., & Wang, J. (2026). AI-driven speech act annotation: Accuracy and reproducibility across ChatGPT, LadderWeb and LLaMA. AI-Linguistica. Linguistic Studies on AI-Generated Texts and Discourses, 3(1). Google Scholar logo with link to Google Scholar
(under review). Machine learning for pragmatic annotation: Comparing supervised and pre-trained models in speech act tagging.
Brown, P., & Levinson, S. C. (1987). Politeness: Some universals in language usage. Cambridge University Press. Google Scholar logo with link to Google Scholar
Canale, M., & Swain, M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 11, 1–47. Google Scholar logo with link to Google Scholar
Chen, X., Lei, Y., & Zhang, Z. (1995). Refusing in Chinese. In G. Kasper (Ed.), Pragmatics of Chinese as native and target language (pp. 119–163). University of Hawaii Press.Google Scholar logo with link to Google Scholar
Cognigni, E. (2014). Migrazione femminile e bisogni di apprendimento in italiano L2 Uno studio di caso presso le donne del ricongiungimento familiare. EL.LE. Educazione Linguistica. Language Education, 3(3), 515–531.Google Scholar logo with link to Google Scholar
Council of Europe. (2020). Common European Framework of Reference for Languages: Companion volume. Council of Europe. [URL]
Cook, V. (1999). Going beyond the native speaker in language teaching. TESOL Quarterly, 33(2), 185–209. Google Scholar logo with link to Google Scholar
Cortés Velásquez, D., & Nuzzo, E. (2022). Declining an invitation: The pragmatics of Italian and Colombian Spanish. In S. Gesuato, G. Salvato, & E. Castello (Eds.), Pragmatic aspects of L2 communication: From awareness through description to assessment (pp. 143–163). Cambridge Scholars Publishing.Google Scholar logo with link to Google Scholar
(2024). Disdir: Cancellations and other refusal strategies [Data set]. ARCHE. [URL]
European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 1191, 1–88. [URL]
Fang, X. (2024). Making refusals via English as a lingua franca: Chinese English speakers’ strategies and sequences. Journal of Pragmatics, 2301, 1–14. Google Scholar logo with link to Google Scholar
Félix-Brasdefer, J. C. (2003). Declining an invitation: A cross-cultural study of pragmatic strategies in American English and Latin American Spanish. Multilingua, 22(3), 225–255. Google Scholar logo with link to Google Scholar
(2006). Linguistic politeness in Mexico: Refusal strategies among male speakers of Mexican Spanish. Journal of Pragmatics, 38(12), 2158–2187. Google Scholar logo with link to Google Scholar
(2008). Sociopragmatic variation: Dispreferred responses in Mexican and Dominican Spanish. Journal of Politeness Research, 4(1), 81–110. Google Scholar logo with link to Google Scholar
(2013). Refusing in L2 Spanish: The effects of the context of learning during a short-term study abroad program. In O. Martí-Arnándiz & P. Salazar-Campillo (Eds.), Refusals in instructional contexts and beyond (Vol. 251, pp. 147–173). Utrecht Studies in Language and Communication. Google Scholar logo with link to Google Scholar
Frescura, M. (1997). Strategie di rifiuto in italiano: Uno studio etnografico. Italica, 74(4), 542–559. Google Scholar logo with link to Google Scholar
Gass, S., & Houck, N. (1999). Interlanguage refusals: A cross-cultural study of Japanese-English. Mouton de Gruyter.Google Scholar logo with link to Google Scholar
Grabowski, K. C. (2016). Assessing pragmatic competence. In D. Tsagari & J. Banerjee (Eds.), Handbook of second language assessment (pp. 165–180). De Gruyter Mouton. Google Scholar logo with link to Google Scholar
Grice, H. P. (1975). Logic and conversation. In P. Cole & J. L. Morgan (Eds.), Syntax and semantics 3: Speech acts (pp. 41–58). Academic Press. Google Scholar logo with link to Google Scholar
House, J. (1996). Developing pragmatic fluency in English as a foreign language: Routines and metapragmatic awareness. Studies in Second Language Acquisition, 18(2), 225–252. Google Scholar logo with link to Google Scholar
Kasper, G., & Roever, C. (2005). Pragmatics in second language learning. In E. Hinkel (Ed.), Handbook of research in second language teaching and learning (pp. 317–334). Routledge.Google Scholar logo with link to Google Scholar
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. Google Scholar logo with link to Google Scholar
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. Google Scholar logo with link to Google Scholar
Nelson, G. L., Carson, J., Al Batal, M., & El Bakary, W. (2002). Cross-cultural pragmatics: Strategy use in Egyptian Arabic and American English refusals. Applied Linguistics, 23(2), 163–189. Google Scholar logo with link to Google Scholar
Nuzzo, E., Brocca, N., & Cortés Velásquez, D. (2025). Refusing and canceling an appointment in Italian as a first language: What models for the assessment of learners’ performances? Italiano Lingua Due, 17(2), 1378–1399.Google Scholar logo with link to Google Scholar
Nuzzo, E., & Cortés Velásquez, D. (2020). Canceling last minute in Italian and Colombian Spanish: A cross-cultural account of pragmalinguistic strategies. Corpus Pragmatics, 41, 1–26. Google Scholar logo with link to Google Scholar
Nuzzo, E., Cortés Velásquez, D., & Brocca, N. (under review). Assessing pragmatic effectiveness in learner Italian: Native speaker judgments of refusals and cancellations in a B1 exam corpus. e-JournALL.
Purpura, J. E. (2004). Assessing grammar. Cambridge University Press. Google Scholar logo with link to Google Scholar
Roever, C. (2011). Testing of second language pragmatics: Past and future. Language Testing, 28(4), 463–481. Google Scholar logo with link to Google Scholar
Searle, J. R. (1977). Speech acts: An essay in the philosophy of language. Cambridge University Press.Google Scholar logo with link to Google Scholar
Sickinger, P., & Renkwitz, K. (2021). How good are native speakers at evaluating pragmatic competence? Native speaker assessments of learner status and appropriate pragmatic behaviour English. Langages, 222(2), 113–131. Google Scholar logo with link to Google Scholar
Su, Y., & Chen, X. (2024). Examining refusal and acceptance sequences employing a data-driven binary rating approach. Research Methods in Applied Linguistics, 3(3), Article 100161. Google Scholar logo with link to Google Scholar
(2011b). The effect of L2 proficiency and study-abroad experience on pragmatic comprehension. Language Learning, 61(3), 904–939. Google Scholar logo with link to Google Scholar
Taguchi, N., & Caprario, M. (2023). L2 pragmatics research and the problem of L1 norms. In C. A. Chapelle (Ed.), The encyclopedia of applied linguistics. John Wiley & Sons. Google Scholar logo with link to Google Scholar
Taguchi, N., & Li, S. (2020). Contrastive pragmatics and second language (L2) pragmatics: Approaches to assessing L2 speech act production. Contrastive Pragmatics, 2(1), 1–23. Google Scholar logo with link to Google Scholar
Verzella, M., & Tommaso, L. (2020). The pragmatics of refusing a request in Italian and American English: A comparative study. Discourse and Interaction, 13(1), 92–112. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue