Evaluation

The differential expression score evaluates how accurately a model recovers the genes that development, or a knockout, actually moved. A differential-expression call is the readout most single-cell experiments are run for and the input to everything downstream, so a model that cannot reproduce it has not reproduced the finding, whatever else it got right.

First, the truth declares its own DE genes. For each gene we test the target against the reference with a two-sided Mann-Whitney U test, control the false discovery rate with the Benjamini-Hochberg procedure at α = 0.05, and apply an effect-size floor that removes changes too small to mean anything biologically:

and split that set by direction into up- and down-regulated genes:

Next, the prediction is given no say in how many genes it names. It is asked for exactly n_u = |U_true| up-genes and n_d = |D_true| down-genes, chosen by its own log fold change, so that naming a thousand genes to catch fifty cannot help:

We then match the two sets sign for sign and normalise to the size of the true set:

Finally, that overlap is scored against a measured null rather than against zero. Highly-expressed genes are mechanically easier to call differentially expressed, and they overlap substantially with the genes that really are developmental DEGs — so a submission that has learned nothing except which genes are highly expressed scored 0.61 on a naive overlap, against 0.17 for the best real baseline. The null is the same construction applied to a prediction that knows only the reference’s own expression level, in both directions, since scaling the reference down games the score as readily as scaling it up:

The directional concordance score measures whether a model moved genes in the direction the real embryo moves them, across the whole transcriptome rather than only the called set. Where DES asks which genes changed, this asks whether the change went the right way — so a model that has the shape of the response right but its magnitude wrong still scores.

First, we take the predicted and observed log fold changes against the reference, and rank-transform both along with the reference’s own expression profile:

Next, letting C be the 3 × 3 Pearson correlation matrix of those three rank vectors, we take the partial correlation of predicted against observed change, controlling for reference expression level:

That control is not a refinement, it is the metric. A real developmental or perturbation response is itself correlated with how highly a gene is already expressed, because highly-expressed genes tend to be the ones under active regulation at that stage. A plain correlation therefore pays a submission for reproducing a relationship that carries no model signal at all: submitting 0.99× the reference scored 0.31 before the control was added. Partialling the reference out leaves only the agreement its own expression level cannot already explain.

A no-change prediction returns 0, as does the degenerate case where the prediction’s entire signal is collinear with reference expression level and there is nothing left to credit.

The perturbation severity score measures how strong the predicted phenotype is relative to the real one. Two knockouts can move the same genes in the same direction and still be different phenotypes — one mild, one severe — so a prediction can score well on DES and DCS and still be wrong about how hard the embryo was hit.

First, we restrict to the truth’s own DE gene set, defined as above. We then fit a zero-intercept least-squares slope of predicted log fold change on observed log fold change, and record how much of the variance that fit explains:

Finally, the score is the signed log of that slope, gated on the fit explaining something and the direction not being inverted:

The gate is why this is a regression slope rather than a ratio of the two response vectors’ norms: norms do not care about direction, so any rotation of a correctly-sized but randomly-directed delta wins a norm ratio outright. A direction unrelated to the truth regresses toward β → 0, and is reported as an honest “undefined” rather than credited with a number.

Maximum mean discrepancy measures whether the predicted population of cells is the same population as the observed one — the right cell states in the right proportions. A stage of development is a mixture of progenitors, differentiating cells and terminal types all at once, not one average cell, and this is the term that holds a submission to that.

First, both the prediction and the target — up to 2000 cells each — are projected into a 30-component PCA basis fitted on the target alone, so that a submission cannot bend the space it is scored in to its own advantage.

Next, for each of five RBF kernel bandwidths γ_s = s · γ₀ with s ∈ {0.25, 0.5, 1, 2, 4}, spread around the target’s own median-heuristic bandwidth γ₀, we compute the squared discrepancy between the two sets of cells:

Finally, we average over the five bandwidths, so the score sees both coarse population shifts and fine within-population structure rather than committing to one length scale:

Dropping the kernel diagonal — the i ≠ i′ and j ≠ j′ conditions — makes this an unbiased estimator. The biased version put a perfect prediction at roughly 2/n rather than 0, a finite-sample artefact that a naive rescaling then mistook for the attainable ceiling.

The co-expression structure score measures whether genes that move together in the real embryo still move together in the prediction. A cell-fate programme is a set of co-regulated genes — a transcription factor and its targets rising in step — and this is the only term in the panel that constrains it. Every other expression metric here is a function of per-gene marginals or lives in a PCA space, so a submission can get every gene’s individual distribution exactly right while scrambling which genes co-vary with which, and pass all of them.

First, we sample 20,000 random gene pairs (i, j) with i ≠ j. For each pair, in each dataset, we take the geostatistical variogram at p = 0.5 — applied across gene pairs instead of spatial lags:

Finally, the score is the mean squared difference between the predicted and observed variograms over those pairs — a direct measurement of whether pairwise gene-gene covariance structure survived the prediction: