Dan Deutsch

We also find that MT systems/LLMs produce better translations than the human translators (at least according to automatic metrics), a result that should be confirmed with a follow up study (hint: future work coming soon!)

We collect machine translations from a variety of different MT systems/LLMs and rank them with automatic metrics. Takeaway: LLMs get the best scores across all languages! Data coming to MTME soon github.com/google-research/m…

🚨New machine translation dataset alert! 🚨We expanded the language coverage of WMT24 from 9 to 55 en->xx language pairs by collecting new reference translations for 46 languages in a dataset called WMT24++ Paper: arxiv.org/abs/2502.12404v1 Data: huggingface.co/datasets/goog…

Running large-scale experiments for building SOTA MT models arxiv.org/pdf/2309.10966

Researching how to improve the quality of human evaluations of text generation arxiv.org/pdf/2404.01474

Using SOTA LLMs for machine translation arxiv.org/pdf/2409.06790

Researching better ways to do metric meta-evaluation arxiv.org/abs/2305.14324

Developing SOTA automatic evaluation metrics arxiv.org/pdf/2410.03983

🌐Exciting News in Machine Translation! 🚀MetricX-23, our SOTA evaluation metric, is now OPEN-SOURCE in PyTorch/Transformers! 🎉There are three model sizes available, all trained on 1m+ human judgments of MT quality! 🔗Code github.com/google-research/m… 🔗Paper www2.statmt.org/wmt23/pdf/20…

@emnlpmeeting I'm running into an error when I try to submit the camera ready version of my paper. All the boxes are checked, but when I hit the submit button, this error message appears. Has anybody else had this problem?

Overall, most metrics do *better* at scoring paragraphs than sentences, possibly because the ground-truth ratings are more stable when they’re aggregated over paragraphs instead of individual sentences. (7/n)

First, to meta-evaluate metrics at the paragraph level, we need paragraph-level human ratings. These datasets are scarce, so we create them from existing sentence-level DA and MQM ratings (which were done in context) by recombining sentences into paragraphs. (5/n)

Interested in document-level MT but have been held back by the lack of automatic metrics? If so, you won't want to miss our new paper! We study the quality of sentence-level metrics on long-form text and augment them with paragraph-level training data. arxiv.org/abs/2308.13506

First, we show that existing Kendall variants have shortcomings related to how they handle ties, and, in some cases, ties can be exploited to game the correlations. A metric could have taken advantage of this to inflate its correlations in the WMT’22 metrics shared task.

LLM-based metrics like GEMBA predict many ties, but the way that ties should be handled in Kendall’s tau for meta-evaluating metrics has been a longstanding issue. We propose an update to the meta-evaluation methodology to handle ties. arxiv.org/pdf/2305.14324.pdf

When we evaluate metrics this way, we find that metrics and humans agree on model quality far less frequently than in the standard setup. This means that the small differences in metric scores reported in papers are not great indicators of whether one model is better than another