The gist: WARP reconstructs the training source mixtures of language models from their weights, achieving mean absolute errors of 0.046 for BERT and 0.104 for GPT-2.
Researchers have developed a framework that reconstructs the data composition of trained language models solely from their published weights. The approach uses model interpolation to simulate missing training trajectories and uncover hidden patterns in weight geometry.
Foundation models are routinely published, but their training recipes — such as the mixture weights that determine how different data sources were combined — typically remain undocumented. This asymmetry makes it difficult for independent researchers to understand the actual training distribution underlying a model. Previous methods for inferring training data, such as membership inference, operate at the sample level and cannot characterize the global composition of the training corpus.
WARP (Weight-Space Analysis for Recovering Training Data Portfolios) addresses this problem through direct reconstruction of domain mixtures from model weights. The approach interpolates between a base model and a fine-tuned model using model merging, generating pseudo-checkpoints that approximate the missing training trajectory. These simulated curves expose a geometric trace of the training data in weight space. From these imprints, WARP extracts geometric features and maps them to domain proportions — either via a parameter-free softmax readout or an MLP projector trained on synthetic mixtures.
In controlled experiments with BERT and GPT-2, WARP achieves average mean absolute errors (MAE) of 0.046 and 0.104 respectively when reconstructing domain mixtures. The system thus outperforms both classical membership inference and a variant with access to the true training trajectory. The method could close transparency gaps and enable verification of the actual data foundation of published models.
Source: arxiv.org · Published 1 July 2026
Lumi AI News — AI-assisted curation in accordance with Article 50 EU AI Act. Paraphrase and classification via Lumi News Pipeline v1.7.3.