Fabian David Schmidt

University of Würzburg

Author Statistics

Papers

Citation

H-Index

i-10 index

Research Trends

Author Order

Document Type

Co-Authors

Goran Glavaš

German Research Centre for Artificial Intelligence

Ivan Vulić

University of Cambridge

Han van der Aa

University of Vienna

Philipp Borchert

Institut d'Economie Scientifique Et de Gestion

Adrian Rebmann

University of Mannheim

Simone Paolo Ponzetto

University of Mannheim

Ivan Vulić

University of Cambridge

Anastasia Ershova

Queen's University Belfast

Nikoleta Yordanova

Leiden University

Aleksandra Khokhlova

Leiden University

Cooperative Institutions

University of Mannheim

IT University of Copenhagen

Tokyo Institute of Technology

Administration for Community Living

American Jewish Committee

Technical University of Darmstadt

Leiden University

Saarland University

Universitat Politècnica de València

Ludwig-Maximilians-Universität München

Author Statistics

Papers

Citation

H-Index

i-10 index

Research Field

SLICER: Sliced Fine-Tuning for Low-Resource Cross-Lingual Transfer for Named Entity Recognition

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (2022)

Fabian David Schmidt Ivan Vulić Goran Glavaš

Large multilingual language models generally demonstrate impressive results in zero-shot cross-lingual transfer, yet often fail to successfully transfer to low-resource languages, even for token-level prediction tasks like named entity recognition (NER). In this work, we introduce a simple yet highly effective approach for improving zero-shot transfer for NER to low-resource languages. We observe that NER fine-tuning in the source language decontextualizes token representations, i.e., tokens increasingly attend to themselves. This increased reliance on token information itself, we hypothesize, triggers a type of overfitting to properties that NE tokens within the source languages share, but are generally not present in NE mentions of target languages. As a remedy, we propose a simple yet very effective sliced fine-tuning for NER (SLICER) that forces stronger token contextualization in the Transformer: we divide the transformed token representations and classifier into disjoint slices that are then independently classified during training. We evaluate SLICER on two standard benchmarks for NER that involve low-resource languages, WikiANN and MasakhaNER, and show that it (i) indeed reduces decontextualization (i.e., extent to which NE tokens attend to themselves), consequently (ii) yielding consistent transfer gains, especially prominent for low-resource target languages distant from the source language.

Named Entity Recognition

Overfitting

10.18653/v1/2022.emnlp-main.740

Cite

Citations (2)

One For All & All For One: Bypassing Hyperparameter Tuning with Model Averaging For Cross-Lingual Transfer

arXiv (Cornell University) (2023)

Fabian David Schmidt Ivan Vulić Goran Glavaš

Multilingual language models enable zero-shot cross-lingual transfer (ZS-XLT): fine-tuned on sizable source-language task data, they perform the task in target languages without labeled instances. The effectiveness of ZS-XLT hinges on the linguistic proximity between languages and the amount of pretraining data for a language. Because of this, model selection based on source-language validation is unreliable: it picks model snapshots with suboptimal target-language performance. As a remedy, some work optimizes ZS-XLT by extensively tuning hyperparameters: the follow-up work then routinely struggles to replicate the original results. Other work searches over narrower hyperparameter grids, reporting substantially lower performance. In this work, we therefore propose an unsupervised evaluation protocol for ZS-XLT that decouples performance maximization from hyperparameter tuning. As a robust and more transparent alternative to extensive hyperparameter tuning, we propose to accumulatively average snapshots from different runs into a single model. We run broad ZS-XLT experiments on both higher-level semantic tasks (NLI, extractive QA) and a lower-level token classification task (NER) and find that conventional model selection based on source-language validation quickly plateaus to suboptimal ZS-XLT performance. On the other hand, our accumulative run-by-run averaging of models trained with different hyperparameters boosts ZS-XLT performance and closely correlates with "oracle" ZS-XLT, i.e., model selection based on target-language validation performance.

Hyperparameter

Hyperparameter Optimization

Transfer of learning

10.48550/arxiv.2310.10532

Cite

Citations (0)

Free Lunch: Robust Cross-Lingual Transfer via Model Checkpoint Averaging

arXiv (Cornell University) (2023)

Fabian David Schmidt Ivan Vulić Goran Glavaš

Massively multilingual language models have displayed strong performance in zero-shot (ZS-XLT) and few-shot (FS-XLT) cross-lingual transfer setups, where models fine-tuned on task data in a source language are transferred without any or with only a few annotated instances to the target language(s). However, current work typically overestimates model performance as fine-tuned models are frequently evaluated at model checkpoints that generalize best to validation instances in the target languages. This effectively violates the main assumptions of "true" ZS-XLT and FS-XLT. Such XLT setups require robust methods that do not depend on labeled target language data for validation and model selection. In this work, aiming to improve the robustness of "true" ZS-XLT and FS-XLT, we propose a simple and effective method that averages different checkpoints (i.e., model snapshots) during task fine-tuning. We conduct exhaustive ZS-XLT and FS-XLT experiments across higher-level semantic tasks (NLI, extractive QA) and lower-level token classification tasks (NER, POS). The results indicate that averaging model checkpoints yields systematic and consistent performance gains across diverse target languages in all tasks. Importantly, it simultaneously substantially desensitizes XLT to varying hyperparameter choices in the absence of target language validation. We also show that checkpoint averaging benefits performance when further combined with run averaging (i.e., averaging the parameters of models fine-tuned over independent runs).

Hyperparameter

Robustness

10.48550/arxiv.2305.16834

Cite

Citations (0)

Don’t Stop Fine-Tuning: On Training Regimes for Few-Shot Cross-Lingual Transfer with Multilingual Language Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (2022)

Fabian David Schmidt Ivan Vulić Goran Glavaš

A large body of recent work highlights the fallacies of zero-shot cross-lingual transfer (ZS-XLT) with large multilingual language models. Namely, their performance varies substantially for different target languages and is the weakest where needed the most: for low-resource languages distant to the source language. One remedy is few-shot transfer (FS-XLT), where leveraging only a few task-annotated instances in the target language(s) may yield sizable performance gains. However, FS-XLT also succumbs to large variation, as models easily overfit to the small datasets. In this work, we present a systematic study focused on a spectrum of FS-XLT fine-tuning regimes, analyzing key properties such as effectiveness, (in)stability, and modularity. We conduct extensive experiments on both higher-level (NLI, paraphrasing) and lower-level tasks (NER, POS), presenting new FS-XLT strategies that yield both improved and more stable FS-XLT across the board. Our findings challenge established FS-XLT methods: e.g., we propose to replace sequential fine-tuning with joint fine-tuning on source and target language instances, offering consistent gains with different number of shots (including resource-rich scenarios). We also show that further gains can be achieved with multi-stage FS-XLT training in which joint multilingual fine-tuning precedes the bilingual source-target specialization.

Overfitting

Modularity

Fine-tuning

Variation (astronomy)

Transfer of learning

10.18653/v1/2022.emnlp-main.736

Cite

Citations (10)

MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching

arXiv (Cornell University) (2025)

Fabian David Schmidt Florian Schneider Chris Biemann Goran Glavaš

Existing multilingual vision-language (VL) benchmarks often only cover a handful of languages. Consequently, evaluations of large vision-language models (LVLMs) predominantly target high-resource languages, underscoring the need for evaluation data for low-resource languages. To address this limitation, we introduce MVL-SIB, a massively multilingual vision-language benchmark that evaluates both cross-modal and text-only topical matching across 205 languages -- over 100 more than the most multilingual existing VL benchmarks encompass. We then benchmark a range of of open-weight LVLMs together with GPT-4o(-mini) on MVL-SIB. Our results reveal that LVLMs struggle in cross-modal topic matching in lower-resource languages, performing no better than chance on languages like N'Koo. Our analysis further reveals that VL support in LVLMs declines disproportionately relative to textual support for lower-resource languages, as evidenced by comparison of cross-modal and text-only topical matching performance. We further observe that open-weight LVLMs do not benefit from representing a topic with more than one image, suggesting that these models are not yet fully effective at handling multi-image tasks. By correlating performance on MVL-SIB with other multilingual VL benchmarks, we highlight that MVL-SIB serves as a comprehensive probe of multilingual VL understanding in LVLMs.

Benchmark (surveying)

10.48550/arxiv.2502.12852

Cite

Citations (0)

Knowledge Distillation vs. Pretraining from Scratch under a Fixed (Computation) Budget

arXiv (Cornell University) (2024)

Minh Duc Bui Fabian David Schmidt Goran Glavaš Katharina von der Wense

Compared to standard language model (LM) pretraining (i.e., from scratch), Knowledge Distillation (KD) entails an additional forward pass through a teacher model that is typically substantially larger than the target student model. As such, KD in LM pretraining materially slows down throughput of pretraining instances vis-a-vis pretraining from scratch. Scaling laws of LM pretraining suggest that smaller models can close the gap to larger counterparts if trained on more data (i.e., processing more tokens)-and under a fixed computation budget, smaller models are able be process more data than larger models. We thus hypothesize that KD might, in fact, be suboptimal to pretraining from scratch for obtaining smaller LMs, when appropriately accounting for the compute budget. To test this, we compare pretraining from scratch against several KD strategies for masked language modeling (MLM) in a fair experimental setup, with respect to amount of computation as well as pretraining data. Downstream results on GLUE, however, do not confirm our hypothesis: while pretraining from scratch performs comparably to ordinary KD under a fixed computation budget, more sophisticated KD strategies, namely TinyBERT (Jiao et al., 2020) and MiniLM (Wang et al., 2023), outperform it by a notable margin. We further find that KD yields larger gains over pretraining from scratch when the data must be repeated under the fixed computation budget.

Scratch

10.48550/arxiv.2404.19319

Cite

Citations (0)

Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages

Fabian David Schmidt Philipp Borchert Ivan Vulić Goran Glavaš

10.18653/v1/2024.findings-emnlp.394

Cite

Citations (0)

Curb EU enthusiasm: how politicisation shapes bureaucratic responsiveness

West European Politics (2024)

Nikoleta Yordanova Aleksandra Khokhlova Anastasia Ershova Fabian David Schmidt Goran Glavaš

International institutions are often challenged for being detached from the citizens. Focusing on the European Union, this article studies whether this is the case and, if so, when the European Commission responds to public opinion when pursuing policy integration. It argues that the Commission has legitimacy incentives encouraging it to be responsive but politicisation can suppress responsiveness by transmitting competing demands on the Commission from the member states' citizens. To test these arguments, the contribution applied automated text analysis to estimate the European Union (EU) authority expansion entailed in the Commission's legislative proposals between 2009 and 2019 and analysed its correspondence with public preferences over EU policy action across the EU states, measured using the Eurobarometer. The results lend support to the hypotheses and suggest that politicisation can undermine the responsiveness of international institutions.

Bureaucracy

10.1080/01402382.2024.2318998

Cite

Citations (3)

SEAGLE: A Platform for Comparative Evaluation of Semantic Encoders for Information Retrieval

Fabian David Schmidt Markus Dietsche Simone Paolo Ponzetto Goran Glavaš

Fabian David Schmidt, Markus Dietsche, Simone Paolo Ponzetto, Goran Glavaš. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations. 2019.

10.18653/v1/d19-3034

Cite

Citations (2)

News Without Borders: Domain Adaptation of Multilingual Sentence Embeddings for Cross-lingual News Recommendation

arXiv (Cornell University) (2024)

Andreea Iana Fabian David Schmidt Goran Glavaš Heiko Paulheim

Rapidly growing numbers of multilingual news consumers pose an increasing challenge to news recommender systems in terms of providing customized recommendations. First, existing neural news recommenders, even when powered by multilingual language models (LMs), suffer substantial performance losses in zero-shot cross-lingual transfer (ZS-XLT). Second, the current paradigm of fine-tuning the backbone LM of a neural recommender on task-specific data is computationally expensive and infeasible in few-shot recommendation and cold-start setups, where data is scarce or completely unavailable. In this work, we propose a news-adapted sentence encoder (NaSE), domain-specialized from a pretrained massively multilingual sentence encoder (SE). To this end, we construct and leverage PolyNews and PolyNewsParallel, two multilingual news-specific corpora. With the news-adapted multilingual SE in place, we test the effectiveness of (i.e., question the need for) supervised fine-tuning for news recommendation, and propose a simple and strong baseline based on (i) frozen NaSE embeddings and (ii) late click-behavior fusion. We show that NaSE achieves state-of-the-art performance in ZS-XLT in true cold-start and few-shot news recommendation.

Domain Adaptation

10.48550/arxiv.2406.12634

Cite

Citations (0)