Self-perceptions of communicative mediation and AI-related self-efficacy in adult EFL learners: A chatbot-supported pilot study

Abstract

This exploratory pilot study investigates adult English as a Foreign Language learners’ self-assessed development following a chatbot-supported pedagogical intervention within an Official Language School in Spain. A 12-item Likert-type questionnaire was administered to 20 adult B2 learners before and after the intervention to gauge shifts across four domains: online interaction self-perception, pluricultural competence self-perception, mediation-related self-perception, and AI-related self-efficacy. The intervention centred on a fictional intercultural scenario in which learners used a chatbot to clarify misunderstandings, infer cultural norms, and formulate contextually appropriate interpretations. Given the ordinal data structure and the restricted sample size, the statistical analysis relied exclusively on non-parametric procedures. Wilcoxon signed-rank tests indicated statistically detectable pre- to post-intervention upward shifts within this sample across all four dimensions as well as in the global self-assessment score. The most pronounced gains were observed in AI-related self-efficacy, while online interaction and mediation-related self-perception also showed comparatively strong upward patterns. At the item level, learners reported particularly marked gains in the management of communicative misunderstandings, audience-targeted information filtering, and the deployment of AI to navigate appropriate behaviour across diverse cultural contexts. While these sample-bound outcomes warrant cautious interpretation due to baseline reliability constraints, the small sample size, and the exclusive reliance on self-report data, they demonstrate the empirical value of self-assessment as a complementary lens. Ultimately, the findings suggest that chatbot-supported pedagogy may have educational value when explicitly embedded within tasks requiring clarification, interpretation, and culturally situated meaning-making, rather than relying solely on decontextualised language practice.

Acknowledgements

The authors thank the participating learners and teaching staff at the Escuela Oficial de Idiomas Do Mundo Lume, in Ayamonte, for their collaboration in the study.

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki and approved by the Universidad of Jaén Ethics Committee, approval code 20240610/JUN.PRY, with approval granted on 10 June 2024.

Funding 

This research was funded by the Regional Government of Andalusia (Junta de Andalucía), through its Consejería de Desarrollo Educativo y Formación Profesional, under the research project AMICIA: Aprendizaje Mediado e Intercultural con Inteligencia Artificial (Project No. PIV-016/25).

CRediT Author Statement

Esther Cores-Bilbao: Conceptualization, Methodology, Software, Validation, Formal Analysis, Investigation, Resources, Data Curation, Writing – original draft, Visualization, Project administration, Funding Acquistion; María-del-Carmen Méndez-García: Conceptualization, Methodology, Software, Validation, Formal Analysis, Investigation, Data Curation, Writing – original draft, Visualization, Supervision; Manuel Célio Conceição: Conceptualization, Methodology, Validation, Formal Analysis, Investigation, Resources, Data Curation, Writing – Reviewing & Editing, Visualization, Supervision.

Disclosure Statement 

The authors reported no potential conflicts of interest.

Generative AI Statement

During the preparation of this work, the authors used ChatGPT-4.0 in order to develop the initial classroom intervention materials, including the fictional cultural scenario, task prompts, and chatbot configuration protocol. The prompts and instructions given to the AI tool were retrieved from ChatGPT and included in the appendices for transparency and replicability. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.