Conference Agenda
Overview and details of the sessions of this conference. Please select a date or location to show only sessions at that day or location. Please select a single session for detailed view (with abstracts and downloads if available).
|
Daily Overview |
| Session | |
|
Donnerstag 1:3: Donnerstag 1:3 – KI in Interaktionsszenarien Location: Hörsaal 2 lecture hall Session Chair: Christof Schöch, Universität Trier | |
    | |
Detecting Literary Evaluations: Can Large Language Models Compete with Human Annotators? Trier Center for Digital Humanities, Trier University, Trier, Germany This study examines to which extent and in which settings Large Language Models (LLMs) can be used to annotate the complex and multi-layered phenomenon of evaluations within literary texts. It uses a gold-standard annotation of German-language fictional narratives published between 1800 and 2015 and compares human annotator agreement to the agreement of LLMs with the gold-standard annotation. The study focuses on ChatGPT, Deepseek, and Llama Sauerkraut-LM using three different prompts and the major vote method. Our results indicate that although LLMs can identify literary evaluations to some degree, their reliability still falls short compared to human annotators. LLMs' performance varies widely across texts, linguistic modernity not being the decisive factor. Clause-level evaluations were more reliably detected by LLMs than noun phrase-level evaluations. The study advances our knowledge of the potential and limitations of LLMs for very complex tasks in the literary domain. | |