Description

Speech translation has been a core focus of IWSLT for years, yet its evaluation remains underexplored. Most existing evaluations assume gold segmentation, an unrealistic scenario for real-world systems. When deferring to automatically segmented speech, conventional text-to-text metrics become less reliable. Despite this, current evaluation practices still rely heavily on these metrics, highlighting the need for more robust and realistic assessment approaches.

This shared task focuses on Quality Estimation for speech translation, a reference-free evaluation of speech translation quality. Participants will assess the quality of translations produced in other IWSLT shared tasks, and system outputs will be evaluated based on their correlation with human judgments.

This year, the task has two tracks:

Speech-to-Text Metrics: quality estimation for speech-to-text translation. We consider two angles:

  1. Speech sample + system translation (text) → score
  2. ASR transcript + system translation (text) → score

Speech-to-Speech Metrics (new in 2027): quality estimation for speech-to-speech translation, where the system output is translated speech:

  • Speech sample + system translation (speech) → score

To evaluate the submissions, we compute correlations with human judgments of quality. We encourage participation in both tracks and in all scenarios, as well as exploring other approaches. The translation segments occur in documents which can also provide additional context to the quality estimation.

We look forward to your submissions!

Contents

Data

Train + Dev Data

Test Data

The test data will be released at the start of the evaluation period.

Example

Speech-to-Text

As an example input, consider the following audio:

and the corresponding testset entry:

{
    "audio": "sample.wav",
    "src_text": "Plans are well underway for races to Mars and the Moon in 1992, by solar sail. The race to Mars is to commemorate Columbus's journey to the New World 500 years ago, and the one to the Moon is to promote the use of solar sails in space exploration.",
    "tgt_text": "Pläne sind gut im Wege für Rennen nach Mars und der Mond in 1992, mit Sonnensegel. Das Rennen zum Mars ist zu Kolumbus' Reise in die Neue Welt vor 500 Jahren zu gedenken, und der eine zum Mond ist den Gebrauch von Sonnensegeln in Weltraum Exploration zu fördern.",
    "score": 71.5
}

The goal is to predict the human score score given the source speech audio, the ASR transcription src_text, and tgt_text. The human score is not 100 because the style of the automatic translation is awkward at places.

Speech-to-Speech

An example for the speech-to-speech track will be added soon.

Baselines

Speech-to-Text

We consider the following quality estimation baselines, which are available on GitHub:

Speech-to-Speech

Baselines for the speech-to-speech track will be added soon.

Evaluation

Quality Estimation models will be evaluated by measuring their correlation with human judgments similar to WMT Metrics Shared Task. For each language pair, we compute:

  • Kendall’s Taub: segment-level measure, akin to Pearson correlation grouped by item. This measures the ability of metrics to select the best translation given a single source.
  • Soft Pairwise Accuracy: system-level measure. This reveals how good the metric is at ranking the participating systems.

The scripts for meta-evaluation of metrics are available publicly on GitHub.

Submission

Submission format and instructions will be announced soon. See Important Dates for deadlines.

Important Dates

The preliminary timeline is below and may be subject to minor changes.

Date Milestone
Jan 1, 2027 Release of shared task training and dev data
Apr 21, 2027 Test set release (evaluation period)
Apr 30, 2027 System submission deadline
May 7, 2027 System paper submission deadline
Jun 7, 2027 Notification of acceptance
Jun 21, 2027 Camera ready deadline
Aug 17-18, 2027 IWSLT conference

Note: All deadlines are 11:59PM UTC-12:00.

Organizers and Contact

Listed in alphabetical order.

  • Erica Cooper, NICT
  • HyoJung Han, University of Maryland
  • Wen-Chin Huang, Nagoya University
  • Dominik Macháček, Charles University
  • Matthias Sperber, Apple
  • Vilém Zouhar, ETH Zurich, vzouhar@ethz.ch (chair)
  • Maike Züfle, Karlsruhe Institute of Technology, maike.zuefle@kit.edu (chair)

Discussion: iwslt-evaluation-campaign@googlegroups.com