How Hidden Traits Spread Between AI Models

In the paper's words

Subliminal learning — the transfer of a teacher model's behavioral traits to a student trained on semantically unrelated outputs such as number sequences — is documented empirically but lacks a quantitative channel model that explains why transfer succeeds within a model family and fails across families. This gap blocks any principled design of filters, training recipes, or audits for the increasingly common practice of training on model-generated data. We formalize subliminal learning as steganographic modulation of the teacher's output distribution: a trait is encoded as a low-amplitude perturbation along a teacher-specific direction in function space, budgeted by a per-token KL divergence D.

The abstract, verbatim, and its opening sentence is the claim. Every line below is quoted from the text.

Numbers it reports

1. At $D = 0.05$ nats the model gives $C = 0.072134$ bits per token, detection at $92.10$ tokens for a known-reference detector at $10^{-2}$ false-alarm tolerance, and a cross-family sample-complexity penalty of a factor of $9$ at $\rho_{\mathrm{cross}} = 0.3$ versus $\rho_{\mathrm{same}} = 0.9$ (factor $81$ at $\rho_{\mathrm{cross}} = 0.1$).

   from Abstract

2. A covert-capacity bound $C = D/\ln 2$ bits per token and a steganalysis sample complexity $n \gtrsim \ln(1/\alpha)/D$, evaluated at $D = 0.05$ nats to $C = 0.072134$ bits per token and $n = 92.10$ tokens at $\alpha = 10^{-2}$ (Section 4, Results in Section 5).

   from 1. Introduction

3. An architecture-specificity theorem: per-sample trait information equals $\rho^2 D$, giving a cross-family sample-complexity penalty of $1/\rho_{\mathrm{cross}}^2$ relative to same-family transfer, computed as a factor of $9$ at $\rho_{\mathrm{cross}} = 0.3$ and $81$ at $\rho_{\mathrm{cross}} = 0.1$ (Section 4).

   from 1. Introduction

4. A filter-survival condition stating that content filtering removes a fraction $f$ of the KL budget only insofar as it projects onto the trait direction, with $D_{\mathrm{eff}} = (1-f)D$ and worked numbers for $f = 0.5$ (Section 4).

   from 1. Introduction

5. A demodulation sample budget $M \geq K \ln 2 / D$ carrier tokens for a $K$-bit behavioral specification, giving $138{,}629$ tokens for $K = 10^4$ bits at $D = 0.05$ nats (Section 4).

   from 1. Introduction

6. Finally, the manifold-modulation capacity paper [10] already derives, for teacher-student transmission channels, the three relations this paper adopts and extends: a per-token capacity $C = D/\ln 2$ bits evaluated at $0.072134$, $0.144270$, and $0.288539$ bits per token for $D = 0.05$, $0.10$, and $0.20$ nats; a steganalysis bound of approximately $\ln(1/\alpha)/D$ tokens, giving $92.10$ tokens at $D = 0.05$ nats and $\alpha = 10^{-2}$; and a demodulation condition $M \geq K \ln 2/D$ giving $138{,}629$ carrier tokens for a $10^4$-bit specification.

   from 2. Background and Related Work

How it is done

1. We derive a covert capacity $C = D/\ln 2$ bits per token, a steganalytic detection threshold $n \gtrsim \ln(1/\alpha)/D$ tokens, and an architecture-specificity condition in which per-sample trait information scales with the squared representational overlap $\rho^2$ between teacher and student directions.

   from Abstract

2. The same protocol transfers misaligned and unsafe behavior, so a pipeline that ingests teacher-generated data can inherit properties that no semantic inspection of the data reveals [3], [4].

   from 1. Introduction

3. Finally, the manifold-modulation capacity paper [10] already derives, for teacher-student transmission channels, the three relations this paper adopts and extends: a per-token capacity $C = D/\ln 2$ bits evaluated at $0.072134$, $0.144270$, and $0.288539$ bits per token for $D = 0.05$, $0.10$, and $0.20$ nats; a steganalysis bound of approximately $\ln(1/\alpha)/D$ tokens, giving $92.10$ tokens at $D = 0.05$ nats and $\alpha = 10^{-2}$; and a demodulation condition $M \geq K \ln 2/D$ giving $138{,}629$ carrier tokens for a $10^4$-bit specification.

   from 2. Background and Related Work

4. This is a theoretical paper.

   from 3. Methods

What would falsify it

1. The model predicts that content filtering preserves the channel whenever the trait direction is orthogonal to the filtered semantic subspace, and it converts empirical observations of brittleness and context dependence into testable quantitative conditions.

   from Abstract

2. With such a model, a curation team could size a filter's residual risk in bits per token, an auditor could state a sample complexity for detection, and a training-recipe designer could predict whether a cross-family student is exposed at all.

   from 1. Introduction

3. In our formalism, related traits correspond to encoding directions with large mutual overlap, so a student demodulating one direction partially reads its neighbors — a prediction the cluster structure confirms qualitatively.

   from 2. Background and Related Work

Where the claim stops

1. The result rests on the following assumptions, each with a distinct sensitivity: Small modulation (A1).

   from 6. Discussion

2. The capacity and sample-complexity numbers at $D = 0.20$ should be read as approximate to within the neglected $O(\varepsilon^3)$ correction.

   from 6. Discussion

3. The $\rho^2$ attenuation assumes the student's SFT update is linear in the trait direction's projection.

   from 6. Discussion

Where to read

1. Abstract227 words · 1 min
2. 1. Introduction740 words · 3 min
3. 2. Background and Related Work980 words · 4 min
4. 3. Methods423 words · 2 min
5. 4. Analysis1,411 words · 6 min
6. 5. Results325 words · 1 min
7. 6. Discussion604 words · 3 min
8. 7. Practical Implications and Engineering Pathways623 words · 3 min
9. 8. Conclusion157 words · 1 min
10. Provenance and AI Involvement240 words · 1 min