Mutant perturbation prediction

T3 · Perturbation — Predict a held-out knockout — expression and 3D coordinates — from wild-type development plus one observed perturbation.

The question

Participants will predict a held-out knockout: the expression and 3D position of the cells in a mutant embryo, given wild-type development plus one observed knockout to learn the shape of a perturbation response from.

A conditional knockout line is months of work per gene, so no lab can make one for every gene it wants to understand. The three knockouts here are ordered by difficulty: training and validation use very specific genes, while the hidden test gene is broadly expressed and its effect correspondingly diffuse.

Scoring is on the perturbation effect, not the absolute state. Predicting no change at all — simply returning the matched wild type — already reaches 0.956 absolute pseudobulk correlation, so nothing but the wild-type-to-mutant change carries any information about whether a model understood the knockout.

Two parallel pairs. On the left a wild-type embryo and, after an arrow, the observed knockout it becomes, both filled. On the right the same wild type and, after an arrow, an empty violet outline where the unseen gene's mutant would be. — The same question asked twice. One knockout is given, to learn the shape of a perturbation response from; the answer is required for a gene the model has never seen.

The same question asked twice. One knockout is given, to learn the shape of a perturbation response from; the answer is required for a gene the model has never seen.

Modality

3D MERFISH from conditionally knocked-out embryos with matched wild-type controls at the same stage. Same schema as Task 2: 500-gene panel, log-normalised expression, per-cell 3D coordinates, cell-type annotation.

Splits

Validation and test ground truth are both withheld. A validation submission returns a leaderboard score, not the answers — those are released only when the final test set is.

Train: Mab21l2 KO @ E9.5; A clear phenotype from a very specific gene.
Validation: Gata4 KO @ E8.75; Also specific, at a different stage. Scored through the leaderboard; the answers are not distributed.
Test: β-catenin KO @ E8.75; Broadly expressed, so its effect is diffuse — the hardest of the three. Hidden until the competition closes.
Reference: Matched WT @ E8.75 and E9.5; Every DE metric is computed against the matched wild type, not against a preceding stage.

Input and output

Input: Wild-type development across the window, plus one observed knockout with its matched control. The model must predict the held-out mutant given the gene identity and the matched wild-type stage data.
Output: An AnnData file for the mutant condition — .X on the 500-gene panel plus obsm["spatial_3D"]. Same contract as Task 2.

The exact file contract and validation rules →

How it is scored

Five metrics rank Task 3: response gene recovery 30%, response direction 25%, response magnitude 25%, cell-state distribution 20%. Tissue shape is deliberately not scored here — it would compare absolute post-knockout geometry, while the meaningful question is whether the perturbation changed shape correctly, and no shift-based spatial term exists yet to sit beside the shift-based expression one.

30%: Response gene recovery
Which genes the knockout actually moved.
25%: Response direction
Whether they moved the way the real knockout moves them.
25%: Response magnitude
How strong the predicted phenotype is, scored on the absolute value.
20%: Cell-state distribution
The same question Tasks 1 and 2 ask, computed the same way — directly against the target, with no wild-type-relative construction.

Every metric, with its derivation →

What the starter kit ships

Transparent reference points, not competitive upper bounds — a floor, a simple method, and a dynamics reference model, all with the exact evaluation code.

wt_identity — the floor. Doing nothing.
shift_transfer — transfer the observed knockout’s pseudobulk response to the target gene.
gene_ko — adds gene-regulatory-network conditioning on the perturbed gene.
perturb_ode — an integrated perturbation field rather than a fixed transferred shift.
dynode_flow — reuses the Task 2 checkpoint through its perturbation path, with no separate training.

Why it’s hard

Absolute expression barely moves: doing nothing already scores 0.956 absolute pseudobulk Pearson. Only the wild-type-to-mutant change carries signal, which is why the panel scores the change and not the absolute state.
One observed perturbation in training. A model has to separate the wild-type developmental trajectory from the genetic effect with very little supervision.
The hidden test gene is broadly expressed, so its effect is diffuse rather than concentrated in a specific programme — the opposite of what the training knockout looks like.
On the published references every model gets the direction of the response wrong, so this is not a task where a small architectural tweak is expected to close the gap.

Common pitfalls

Copying the training knockout’s delta onto the test gene. It recovers a share of the effect and gets the direction wrong, which the panel scores explicitly.
Optimising absolute expression. The floor already wins that comparison, by construction.
Reading the shape group as a ranking signal here. For Task 3 it is diagnostic for now: it compares absolute post-knockout shape, while the meaningful question is whether the perturbation changed shape correctly, and no shift-based spatial term exists yet.
Ignoring the stage difference between the training knockout and the evaluation ones.

Other tasks

T1Temporal gene-expression distribution prediction

Given stages before a target time, predict the gene-expression distribution — a set of cells — at a future stage. Tests temporal extrapolation.

T2Spatial-temporal multiscale prediction

Predict future gene expression and 3D spatial location jointly, across molecular, cellular and spatial scales. Evaluated at two organ scopes.

The task documents, exact metric implementations and executed tutorials live in the baselines repo ↗; where this page and the repo differ, the repo is authoritative. Otherwise reach the organisers at [email protected].