CAICCAIC: Center for Artificial Intelligence Challenge on Conversational AI Correctness
The goal of this competition is to develop Natural Language Understanding models that are robust against speech recognition errors. Regardless of near-human accuracy of Automatic Speech Recognition in general-purpose transcription tasks, speech recognition errors can significantly deteriorate the performance of a Natural Language Understanding model that follows the speech-to-text module in a virtual assistant. The problem is even more apparent when an ASR system from an external vendor is used as an integral part of a conversational system without any further adaptation. The contestants are expected to develop NLU models that maintain satisfactory performance regardless of the presence of ASR errors in the input.
The data for the competition consist of natural language utterances along with semantic frames that represent the commands targeted at a virtual assistant. The detailed specification of the task and the description of the evaluation procedure are available here.
The solutions for the task are to be submitted via the Gonito platform challenge. In order to be included in the final ranking the participants are expected to provide the report that describes their solution. The reports should conform to the requirements for papers submitted to FedCSIS conference and should not exceed 4 pages.
Special session at FedCSIS 2023
The authors of selected submissions will be invited to prepare the extended versions of their reports for publication in the conference proceedings and presentation at FedCSIS 2023. The selection will be made by the Organizing Committee on the basis of final evaluation results, quality of the submitted reports and originality of the presented methods.
- Feb 13, 2023: Training data available (English)
- Feb 20, 2023: Training data available (other languages)
- May 17, 2023: Test data available
- June 14, 2023: Deadline for submitting the results
- June 16, 2023: Announcement of the final results, sending invitations for submitting papers
- July 9, 2023: Deadline for submitting invited papers
- July 16, 2023: Author notification
- July 31, 2023: Final paper submission, registration
- Sept 20, 2023: Challenges in Natural Language Processing Symposium
- Marek Kubis, Adam Mickiewicz University, Poland
- Paweł Skórzewski, Adam Mickiewicz University, Poland
- Marcin Sowański, Samsung Research Poland
- Tomasz Ziętkiewicz, Samsung Research Poland