JOKER

JOKER

•Home •Tasks •CLEF program •Partners •Contest •Contact Us •CLEF 2022 •CLEF 2023 •CLEF 2024


CLEF 2025 JOKER Track:

Automatic Humour Analysis

Abstract

Over the last three years, the JOKER Lab series at CLEF has gathered an active community of researchers in natural language processing and information retrieval to collaborate on non-literal use of language in text. Such language can be a challenge for AI systems, but also sometimes for humans, as it requires understanding implicit cultural references and unorthodox interactions between form and meaning.

Introduction

Humour plays a vital role in social interaction. Understanding it, however, can be challenging for humans, often requiring a good grasp of cultural references and double meanings. State-of-the-art artificial intelligence (AI), natural language processing (NLP), and information retrieval (IR) models also remain largely impervious to humour or other non-literal meaning aspects of texts. This is especially true for tasks like wordplay detection or analysis, which rely on the surface structure (orthography or pronunciation) of a word; such surface-level features are not directly captured in the deep semantic embeddings of modern AI models. They also cannot be captured by current pre-training models based on next-word prediction objectives, which tend to learn literal, statistically likely patterns in language rather than nuanced, non-literal meanings often associated with humour, wordplay, or sarcasm.

The JOKER Lab, now in its fourth year, aims to bring together social and computer scientists to create reusable test collections featuring wordplay and humour, and to foster work on automatic humour analysis.

Tasks

Use Case

Humor is one of the most important aspects of social interaction. Despite significant advances in AI and NLP, understanding humor remains a challenge, as it often involves grasping implicit cultural references and/or double meanings.

The goal of the JOKER lab is to bring together linguists and computer scientists to create reusable test collections that foster work on automatic humor analysis. To encourage research in humor-aware information retrieval, JOKER 2024 introduced a new task aimed at retrieving short humorous texts from a document collection.

The intended use case is to search for a joke on a specific topic. This can be useful for:

Important dates

How to participate

In order to participate, you should sign up at the CLEF website. The registration closes on April 25, 2025.

All team members should join the JOKER mailing list: https://groups.google.com/u/4/g/joker-project.

The data will be made available to all registered participants.

How to Cite

If you extend or use this work, please cite the paper where it was introduced:

Ermakova, L., Bosser, AG., Miller, T., Campos, R. (2025). CLEF 2025 JOKER Lab: Humour in the Machine. In: Hauff, C., et al. Advances in Information Retrieval. ECIR 2025. Lecture Notes in Computer Science, vol 15576. Springer, Cham. https://doi.org/10.1007/978-3-031-88720-8_59.

Liana Ermakova, Anne-Gwenn Bosser, Adam Jatowt, and Tristan Miller. 2023. The JOKER Corpus: English-French Parallel Data for Multilingual Wordplay Recognition. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ‘23). Association for Computing Machinery, New York, NY, USA, 2796–2806. https://doi.org/10.1145/3539618.3591885

This project has received a government grant managed by the National Research Agency under the program "Investissements d'avenir" integrated into France 2030, with the Reference ANR-19-GURE-0001. It was also financed by National Funds through the FCT - Fundação para a Ciência e a Tecnologia, I.P. (Portuguese Foundation for Science and Technology) within the project StorySense, with reference 2022.09312.PTDC (DOI 10.54499/2022.09312.PTDC).

JOKER is supported by The Human Science Institute in Brittany (MSHB)