Published in the Italian Journal of Computational Linguistics (IJCoL), vol. 12, n. 1 (June 2026), Accademia University Press, DOI 10.17454/ijcol121.02.
The paper describes CALAMITA, a benchmarking initiative coordinated by the Italian Association for Computational Linguistics in which more than 80 contributors built over 20 tasks and almost 100 subtasks to evaluate LLMs in Italian. It reports results for four open-weight LLMs and discusses lessons on fine-grained metrics, shared evaluation pipelines and community involvement.
Authors: Malvina Nissim, Danilo Croce, Viviana Patti, Pierpaolo Basile and 77 other authors, including Kamyar Zeinalipour and Andrea Zugarini.