Why a model behaves differently at the next hospital

Problem

Prognostic models are often built from several data sources at once: hospitals in a multi-center study, registries, separate trials. The relationships those sources produce rarely agree. Standard meta-analytic measures such as coefficient-level tau-squared say how far apart they are but not why, and the two possible answers point in opposite directions. If the sources enrolled different kinds of patients, one pooled model is still defensible. If comparable patients follow different relationships, pooling averages incompatible effects into one that fits nobody.

Approach

At the study level the two causes look identical, so the question has to be asked one patient at a time. An autoencoder trained on reconstruction, local prognostic, and native-site terms learns a low-dimensional representation of patient characteristics. Ridge-stabilized local regressions then estimate a coefficient vector at each patient’s position in that space, using only observations from that patient’s own site. Kernel-ridge surfaces smooth those coefficients, so two sites can be compared at matched patient profiles. Partitioning the surfaces separates a cross-site position component from a site-specific context component. The method then projects that split onto the outcome scale and reports summaries per observation and per site.

Outcome

I applied the method to a clinical trial with two participating centers. In the leading coordinates of the learned representation, the gap between the two sites’ coefficients came mostly from context rather than from patient mix. Comparable patients did not respond the same way at both centers. Carried back to the outcome scale, the difference between the sites concentrates in context too, and case mix accounts for close to none of it. Rerunning the pipeline with the site labels shuffled at random cut that contextual signal by more than a factor of ten, which is what should happen if the labels carry nothing.