Evaluation

The differential expression score evaluates how accurately a model recovers the genes that development, or a knockout, actually moved. A differential-expression call is the readout most single-cell experiments are run for and the input to everything downstream, so a model that cannot reproduce it has not reproduced the finding, whatever else it got right.

First, the truth declares its own DE genes. For each gene we test the target against the reference with a two-sided Mann-Whitney U test, control the false discovery rate with the Benjamini-Hochberg procedure at α = 0.05, and apply an effect-size floor that removes changes too small to mean anything biologically:

and split that set by direction into up- and down-regulated genes:

Next, the prediction is given no say in how many genes it names. It is asked for exactly n_u = |U_true| up-genes and n_d = |D_true| down-genes, chosen by its own log fold change, so that naming a thousand genes to catch fifty cannot help:

We then match the two sets sign for sign and normalise to the size of the true set:

Finally, that overlap is scored against a measured null rather than against zero. Highly-expressed genes are mechanically easier to call differentially expressed, and they overlap substantially with the genes that really are developmental DEGs — so a submission that has learned nothing except which genes are highly expressed scored 0.61 on a naive overlap, against 0.17 for the best real baseline. The null is the same construction applied to a prediction that knows only the reference’s own expression level, in both directions, since scaling the reference down games the score as readily as scaling it up:

The directional concordance score measures whether a model moved genes in the direction the real embryo moves them, across the whole transcriptome rather than only the called set. Where DES asks which genes changed, this asks whether the change went the right way — so a model that has the shape of the response right but its magnitude wrong still scores.

First, we take the predicted and observed log fold changes against the reference, and rank-transform both along with the reference’s own expression profile:

Next, letting C be the 3 × 3 Pearson correlation matrix of those three rank vectors, we take the partial correlation of predicted against observed change, controlling for reference expression level:

That control is not a refinement, it is the metric. A real developmental or perturbation response is itself correlated with how highly a gene is already expressed, because highly-expressed genes tend to be the ones under active regulation at that stage. A plain correlation therefore pays a submission for reproducing a relationship that carries no model signal at all: submitting 0.99× the reference scored 0.31 before the control was added. Partialling the reference out leaves only the agreement its own expression level cannot already explain.

A no-change prediction returns 0, as does the degenerate case where the prediction’s entire signal is collinear with reference expression level and there is nothing left to credit.

Maximum mean discrepancy measures whether the predicted population of cells is the same population as the observed one — the right cell states in the right proportions. A stage of development is a mixture of progenitors, differentiating cells and terminal types all at once, not one average cell, and this is the term that holds a submission to that.

First, both the prediction and the target — up to 2000 cells each — are projected into a 30-component PCA basis fitted on the target alone, so that a submission cannot bend the space it is scored in to its own advantage.

Next, for each of five RBF kernel bandwidths γ_s = s · γ₀ with s ∈ {0.25, 0.5, 1, 2, 4}, spread around the target’s own median-heuristic bandwidth γ₀, we compute the squared discrepancy between the two sets of cells:

Finally, we average over the five bandwidths, so the score sees both coarse population shifts and fine within-population structure rather than committing to one length scale:

Dropping the kernel diagonal — the i ≠ i′ and j ≠ j′ conditions — makes this an unbiased estimator. The biased version put a perfect prediction at roughly 2/n rather than 0, a finite-sample artefact that a naive rescaling then mistook for the attainable ceiling.

The co-expression structure score measures whether genes that move together in the real embryo still move together in the prediction. A cell-fate programme is a set of co-regulated genes — a transcription factor and its targets rising in step — and this is the only term in the panel that constrains it. Every other expression metric here is a function of per-gene marginals or lives in a PCA space, so a submission can get every gene’s individual distribution exactly right while scrambling which genes co-vary with which, and pass all of them.

First, we sample 20,000 random gene pairs (i, j) with i ≠ j. For each pair, in each dataset, we take the geostatistical variogram at p = 0.5 — applied across gene pairs instead of spatial lags:

Finally, the score is the mean squared difference between the predicted and observed variograms over those pairs — a direct measurement of whether pairwise gene-gene covariance structure survived the prediction:

The shape distribution distance measures whether the predicted tissue has the right form. An embryo is a shape, and a model that produces the right cells in the wrong arrangement has not modelled morphogenesis — but comparing two embryos imaged in independent, arbitrary frames needs a descriptor that does not depend on the frame at all.

First, for each point cloud we sample 2 × 10⁵ random point pairs and normalise their distance by that cloud’s own RMS radius, giving the distribution of how far apart pairs of cells inside the tissue tend to be, relative to its own size:

Finally, the score is the earth-mover’s distance between those two empirical distributions:

Because it only ever uses distances within a single cloud, this term needs no registration step, no shared frame, and no cell correspondence between prediction and target in order to be defined — so it carries zero registration risk. It was measured to be monotone in true developmental distance, robust to a 16-fold cell-count difference, and to rank a real morphological prediction above a featureless blob rather than tying with one.

The occupancy Dice score measures whether the predicted tissue occupies the same region of space as the real one — the volumetric counterpart to the shape distribution above, which compares form but not the space it fills.

First, both clouds are canonicalised — centred, PCA-aligned and rescaled by RMS radius — and voxelised on the same fixed 16³ grid spanning [−3, 3] RMS radii, giving the set of occupied voxels:

Finally, we take the Dice overlap of those two sets, maximised over the four sign flips with determinant +1 — proper rotations only, since PCA fixes each axis but not its sign:

Restricting to proper rotations means the score does not silently credit a mirror image — but neither can it detect one. Laterality is a stated blind spot of every shape term here: a mirrored embryo scores identically to a correct one.

The tissue scale ratio measures growth: whether the embryo or organ has reached the right physical size at the target stage. Getting the form right at the wrong scale is a different failure from getting the form wrong, and development is as much about how much tissue there is as about how it is arranged.

Size is taken as the root-mean-square distance of cells from their centroid — rotation-invariant by construction, unlike a bounding-box extent — and the score is the signed log of the ratio between prediction and target:

The neighborhood fidelity score measures whether each cell sits among the right neighbours. Tissue identity is local: what makes a region cardiac outflow tract is not only which genes its cells express but which other cells they are next to, since signalling, boundaries and niches are all defined at the scale of a neighbourhood. This is the term that actually exercises the joint expression-and-position submission Tasks 2 and 3 ask for, rather than scoring expression and shape as two unrelated things.

First, for every cell we take its spatial-neighbourhood pseudobulk — the mean expression of its 15 nearest neighbours in space:

Finally, we compare the two resulting sets of neighbourhood profiles with the same unbiased multi-kernel discrepancy defined above:

Because this compares the distributions of neighbourhood-averaged profiles, it never assumes that predicted cell i is true cell i — it is coupling-independent by construction, so a submission is not required to reproduce any particular cell-to-cell correspondence with the target.